2023-07-10T16:44:15.967 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-10T16:44:15.969 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} --owner scheduled_yuriw@teuthology --name yuriw-2023-07-10_14:39:39-orch-wip-yuri8-testing-2023-07-06-0909-reef-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-07-10_14:39:39-orch-wip-yuri8-testing-2023-07-06-0909-reef-distro-default-smithi/7332179 -- /home/teuthworker/archive/yuriw-2023-07-10_14:39:39-orch-wip-yuri8-testing-2023-07-06-0909-reef-distro-default-smithi/7332179/orig.config.yaml 2023-07-10T16:44:16.014 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-10T16:44:16.108 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-07-10_14:39:39-orch-wip-yuri8-testing-2023-07-06-0909-reef-distro-default-smithi/7332179 branch: wip-yuri8-testing-2023-07-06-0909-reef description: orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} email: yweinste@redhat.com first_in_suite: false job_id: '7332179' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-07-10_14:39:39-orch-wip-yuri8-testing-2023-07-06-0909-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri8-testing-2023-07-06-0909-reef ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 8507faa7ce284e46218d481e3ee9c411ac3dc32c ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 8507faa7ce284e46218d481e3ee9c411ac3dc32c workunit: branch: wip-yuri8-testing-2023-07-06-0909-reef sha1: 8507faa7ce284e46218d481e3ee9c411ac3dc32c owner: scheduled_yuriw@teuthology priority: 95 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 2303 sha1: 8507faa7ce284e46218d481e3ee9c411ac3dc32c sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: wip-yuri8-testing-2023-07-06-0909-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8507faa7ce284e46218d481e3ee9c411ac3dc32c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8507faa7ce284e46218d481e3ee9c411ac3dc32c targets: smithi116.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= smithi192.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= tasks: - cephadm: allow_ptrace: false avoid_pacific_features: true cephadm_branch: v16.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v16.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b53f236c1077f1d295baafb9e0ca9a4b447019d7 timestamp: 2023-07-10_14:39:39 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-10T16:44:16.109 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8507faa7ce284e46218d481e3ee9c411ac3dc32c/qa; will attempt to use it 2023-07-10T16:44:16.109 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8507faa7ce284e46218d481e3ee9c411ac3dc32c/qa/tasks 2023-07-10T16:44:16.110 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-10T16:44:16.112 INFO:teuthology.task.internal:Checking packages... 2023-07-10T16:44:16.142 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '8507faa7ce284e46218d481e3ee9c411ac3dc32c' 2023-07-10T16:44:16.142 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-10T16:44:16.142 INFO:teuthology.packaging:ref: None 2023-07-10T16:44:16.143 INFO:teuthology.packaging:tag: None 2023-07-10T16:44:16.143 INFO:teuthology.packaging:branch: wip-yuri8-testing-2023-07-06-0909-reef 2023-07-10T16:44:16.143 INFO:teuthology.packaging:sha1: 8507faa7ce284e46218d481e3ee9c411ac3dc32c 2023-07-10T16:44:16.143 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri8-testing-2023-07-06-0909-reef 2023-07-10T16:44:16.282 INFO:teuthology.task.internal:Found packages for ceph version 18.1.2-30-g8507faa7-1focal 2023-07-10T16:44:16.282 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-10T16:44:16.289 INFO:teuthology.task.internal:no buildpackages task found 2023-07-10T16:44:16.290 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-10T16:44:16.318 INFO:teuthology.task.internal:Saving configuration 2023-07-10T16:44:16.330 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-10T16:44:16.343 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-10T16:44:16.368 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-10_14:39:39-orch-wip-yuri8-testing-2023-07-06-0909-reef-distro-default-smithi/7332179', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-10 16:38:16.446351', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-07-10T16:44:16.393 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-10_14:39:39-orch-wip-yuri8-testing-2023-07-06-0909-reef-distro-default-smithi/7332179', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-10 16:38:16.444956', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-07-10T16:44:16.394 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-10T16:44:16.400 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2023-07-10T16:44:16.401 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2023-07-10T16:44:16.401 INFO:teuthology.run_tasks:Running task console_log... 2023-07-10T16:44:16.504 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb8b6521430>, signals=[15]) 2023-07-10T16:44:16.506 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-10T16:44:16.514 INFO:teuthology.task.internal:Opening connections... 2023-07-10T16:44:16.514 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2023-07-10T16:44:16.516 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-10T16:44:16.569 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2023-07-10T16:44:16.570 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-10T16:44:16.619 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-10T16:44:16.631 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2023-07-10T16:44:16.647 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2023-07-10T16:44:16.648 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2023-07-10T16:44:16.654 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="Ubuntu" 2023-07-10T16:44:16.654 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-07-10T16:44:16.655 INFO:teuthology.orchestra.run.smithi116.stdout:ID=ubuntu 2023-07-10T16:44:16.655 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE=debian 2023-07-10T16:44:16.655 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-07-10T16:44:16.655 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="20.04" 2023-07-10T16:44:16.655 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://www.ubuntu.com/" 2023-07-10T16:44:16.655 INFO:teuthology.orchestra.run.smithi116.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-07-10T16:44:16.655 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-07-10T16:44:16.656 INFO:teuthology.orchestra.run.smithi116.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-07-10T16:44:16.656 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_CODENAME=focal 2023-07-10T16:44:16.656 INFO:teuthology.orchestra.run.smithi116.stdout:UBUNTU_CODENAME=focal 2023-07-10T16:44:16.657 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2023-07-10T16:44:16.692 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2023-07-10T16:44:16.697 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2023-07-10T16:44:16.697 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2023-07-10T16:44:16.702 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="Ubuntu" 2023-07-10T16:44:16.703 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-07-10T16:44:16.703 INFO:teuthology.orchestra.run.smithi192.stdout:ID=ubuntu 2023-07-10T16:44:16.703 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE=debian 2023-07-10T16:44:16.703 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-07-10T16:44:16.703 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="20.04" 2023-07-10T16:44:16.703 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://www.ubuntu.com/" 2023-07-10T16:44:16.703 INFO:teuthology.orchestra.run.smithi192.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-07-10T16:44:16.704 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-07-10T16:44:16.704 INFO:teuthology.orchestra.run.smithi192.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-07-10T16:44:16.704 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_CODENAME=focal 2023-07-10T16:44:16.704 INFO:teuthology.orchestra.run.smithi192.stdout:UBUNTU_CODENAME=focal 2023-07-10T16:44:16.704 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2023-07-10T16:44:16.734 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-10T16:44:16.745 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-10T16:44:16.771 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-10T16:44:16.772 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2023-07-10T16:44:16.775 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2023-07-10T16:44:16.781 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-10T16:44:16.806 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-10T16:44:16.806 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2023-07-10T16:44:16.808 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2023-07-10T16:44:16.839 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-10T16:44:16.902 INFO:teuthology.run_tasks:Running task kernel... 2023-07-10T16:44:16.923 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-07-10T16:44:16.923 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-07-10T16:44:16.924 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-10T16:44:16.925 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-10T16:44:16.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-10T16:44:16.930 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2023-07-10T16:44:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-10T16:44:16.932 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2023-07-10T16:44:16.938 INFO:teuthology.orchestra.run.smithi116.stdout:5.4.0-124-generic 2023-07-10T16:44:16.938 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.4.0-124-generic 2023-07-10T16:44:16.939 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get clean 2023-07-10T16:44:16.939 INFO:teuthology.orchestra.run.smithi192.stdout:5.4.0-124-generic 2023-07-10T16:44:16.940 INFO:teuthology.task.kernel:Running kernel on smithi192: 5.4.0-124-generic 2023-07-10T16:44:16.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get clean 2023-07-10T16:44:17.084 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get update 2023-07-10T16:44:17.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get update 2023-07-10T16:44:17.278 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-07-10T16:44:17.279 INFO:teuthology.orchestra.run.smithi192.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-07-10T16:44:17.279 INFO:teuthology.orchestra.run.smithi192.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-07-10T16:44:17.291 INFO:teuthology.orchestra.run.smithi116.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-07-10T16:44:17.346 INFO:teuthology.orchestra.run.smithi192.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-07-10T16:44:17.401 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-07-10T16:44:17.483 INFO:teuthology.orchestra.run.smithi116.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-07-10T16:44:17.571 INFO:teuthology.orchestra.run.smithi116.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [620 kB] 2023-07-10T16:44:17.651 INFO:teuthology.orchestra.run.smithi116.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,304 kB] 2023-07-10T16:44:17.667 INFO:teuthology.orchestra.run.smithi192.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,304 kB] 2023-07-10T16:44:17.690 INFO:teuthology.orchestra.run.smithi116.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [366 kB] 2023-07-10T16:44:17.693 INFO:teuthology.orchestra.run.smithi116.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.0 kB] 2023-07-10T16:44:17.694 INFO:teuthology.orchestra.run.smithi116.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [31.8 kB] 2023-07-10T16:44:17.694 INFO:teuthology.orchestra.run.smithi116.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,979 kB] 2023-07-10T16:44:17.708 INFO:teuthology.orchestra.run.smithi116.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [277 kB] 2023-07-10T16:44:17.709 INFO:teuthology.orchestra.run.smithi116.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2023-07-10T16:44:17.710 INFO:teuthology.orchestra.run.smithi116.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [854 kB] 2023-07-10T16:44:17.715 INFO:teuthology.orchestra.run.smithi116.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [604 kB] 2023-07-10T16:44:17.754 INFO:teuthology.orchestra.run.smithi116.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [176 kB] 2023-07-10T16:44:17.812 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,686 kB] 2023-07-10T16:44:17.813 INFO:teuthology.orchestra.run.smithi192.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [620 kB] 2023-07-10T16:44:17.818 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [366 kB] 2023-07-10T16:44:17.821 INFO:teuthology.orchestra.run.smithi192.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.0 kB] 2023-07-10T16:44:17.822 INFO:teuthology.orchestra.run.smithi192.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [31.8 kB] 2023-07-10T16:44:17.822 INFO:teuthology.orchestra.run.smithi192.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,979 kB] 2023-07-10T16:44:17.825 INFO:teuthology.orchestra.run.smithi116.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-07-10T16:44:17.836 INFO:teuthology.orchestra.run.smithi192.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [277 kB] 2023-07-10T16:44:17.838 INFO:teuthology.orchestra.run.smithi192.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2023-07-10T16:44:17.838 INFO:teuthology.orchestra.run.smithi192.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [854 kB] 2023-07-10T16:44:17.841 INFO:teuthology.orchestra.run.smithi116.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [18.6 kB] 2023-07-10T16:44:17.842 INFO:teuthology.orchestra.run.smithi116.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [29.3 kB] 2023-07-10T16:44:17.844 INFO:teuthology.orchestra.run.smithi192.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [604 kB] 2023-07-10T16:44:17.844 INFO:teuthology.orchestra.run.smithi116.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,388 B] 2023-07-10T16:44:17.844 INFO:teuthology.orchestra.run.smithi116.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [6,120 B] 2023-07-10T16:44:17.845 INFO:teuthology.orchestra.run.smithi116.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [620 B] 2023-07-10T16:44:17.883 INFO:teuthology.orchestra.run.smithi192.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [176 kB] 2023-07-10T16:44:17.914 INFO:teuthology.orchestra.run.smithi192.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [851 kB] 2023-07-10T16:44:17.923 INFO:teuthology.orchestra.run.smithi192.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [448 kB] 2023-07-10T16:44:17.928 INFO:teuthology.orchestra.run.smithi192.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.9 kB] 2023-07-10T16:44:17.929 INFO:teuthology.orchestra.run.smithi192.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,089 kB] 2023-07-10T16:44:17.943 INFO:teuthology.orchestra.run.smithi192.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [33.2 kB] 2023-07-10T16:44:17.944 INFO:teuthology.orchestra.run.smithi192.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [292 kB] 2023-07-10T16:44:17.945 INFO:teuthology.orchestra.run.smithi192.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2023-07-10T16:44:17.967 INFO:teuthology.orchestra.run.smithi116.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [851 kB] 2023-07-10T16:44:17.969 INFO:teuthology.orchestra.run.smithi192.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [18.6 kB] 2023-07-10T16:44:17.970 INFO:teuthology.orchestra.run.smithi192.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,388 B] 2023-07-10T16:44:17.971 INFO:teuthology.orchestra.run.smithi192.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [29.3 kB] 2023-07-10T16:44:17.973 INFO:teuthology.orchestra.run.smithi192.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [6,120 B] 2023-07-10T16:44:17.974 INFO:teuthology.orchestra.run.smithi192.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [620 B] 2023-07-10T16:44:17.980 INFO:teuthology.orchestra.run.smithi192.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [735 kB] 2023-07-10T16:44:18.103 INFO:teuthology.orchestra.run.smithi192.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,082 kB] 2023-07-10T16:44:18.126 INFO:teuthology.orchestra.run.smithi192.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [258 kB] 2023-07-10T16:44:18.131 INFO:teuthology.orchestra.run.smithi192.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.2 kB] 2023-07-10T16:44:18.131 INFO:teuthology.orchestra.run.smithi192.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,628 B] 2023-07-10T16:44:18.131 INFO:teuthology.orchestra.run.smithi192.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [31.5 kB] 2023-07-10T16:44:18.132 INFO:teuthology.orchestra.run.smithi192.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [8,032 B] 2023-07-10T16:44:18.132 INFO:teuthology.orchestra.run.smithi192.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [688 B] 2023-07-10T16:44:18.132 INFO:teuthology.orchestra.run.smithi192.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-07-10T16:44:18.133 INFO:teuthology.orchestra.run.smithi192.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-07-10T16:44:18.133 INFO:teuthology.orchestra.run.smithi192.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-07-10T16:44:18.168 INFO:teuthology.orchestra.run.smithi192.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-07-10T16:44:18.207 INFO:teuthology.orchestra.run.smithi116.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,686 kB] 2023-07-10T16:44:18.207 INFO:teuthology.orchestra.run.smithi192.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-07-10T16:44:18.212 INFO:teuthology.orchestra.run.smithi192.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-07-10T16:44:18.370 INFO:teuthology.orchestra.run.smithi116.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [448 kB] 2023-07-10T16:44:18.383 INFO:teuthology.orchestra.run.smithi116.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.9 kB] 2023-07-10T16:44:18.383 INFO:teuthology.orchestra.run.smithi116.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,089 kB] 2023-07-10T16:44:18.436 INFO:teuthology.orchestra.run.smithi116.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [33.2 kB] 2023-07-10T16:44:18.437 INFO:teuthology.orchestra.run.smithi116.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [292 kB] 2023-07-10T16:44:18.442 INFO:teuthology.orchestra.run.smithi116.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2023-07-10T16:44:18.607 INFO:teuthology.orchestra.run.smithi116.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,082 kB] 2023-07-10T16:44:19.236 INFO:teuthology.orchestra.run.smithi116.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [735 kB] 2023-07-10T16:44:19.275 INFO:teuthology.orchestra.run.smithi116.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [258 kB] 2023-07-10T16:44:19.288 INFO:teuthology.orchestra.run.smithi116.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.2 kB] 2023-07-10T16:44:19.290 INFO:teuthology.orchestra.run.smithi116.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,628 B] 2023-07-10T16:44:19.290 INFO:teuthology.orchestra.run.smithi116.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [31.5 kB] 2023-07-10T16:44:19.292 INFO:teuthology.orchestra.run.smithi116.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [8,032 B] 2023-07-10T16:44:19.292 INFO:teuthology.orchestra.run.smithi116.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [688 B] 2023-07-10T16:44:19.293 INFO:teuthology.orchestra.run.smithi116.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-07-10T16:44:19.318 INFO:teuthology.orchestra.run.smithi116.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-07-10T16:44:19.320 INFO:teuthology.orchestra.run.smithi116.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-07-10T16:44:19.485 INFO:teuthology.orchestra.run.smithi116.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-07-10T16:44:19.669 INFO:teuthology.orchestra.run.smithi116.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-07-10T16:44:19.691 INFO:teuthology.orchestra.run.smithi116.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-07-10T16:44:21.655 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 16.3 MB in 2s (6,743 kB/s) 2023-07-10T16:44:21.657 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 16.3 MB in 3s (6,471 kB/s) 2023-07-10T16:44:23.004 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2023-07-10T16:44:23.023 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2023-07-10T16:44:23.030 DEBUG:teuthology.orchestra.run.smithi192:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-07-10T16:44:23.048 DEBUG:teuthology.orchestra.run.smithi116:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-07-10T16:44:23.100 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2023-07-10T16:44:23.117 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2023-07-10T16:44:23.275 INFO:teuthology.orchestra.run.smithi192.stdout:Building dependency tree... 2023-07-10T16:44:23.275 INFO:teuthology.orchestra.run.smithi192.stdout:Reading state information... 2023-07-10T16:44:23.312 INFO:teuthology.orchestra.run.smithi116.stdout:Building dependency tree... 2023-07-10T16:44:23.313 INFO:teuthology.orchestra.run.smithi116.stdout:Reading state information... 2023-07-10T16:44:23.442 INFO:teuthology.orchestra.run.smithi192.stdout:The following additional packages will be installed: 2023-07-10T16:44:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: linux-generic linux-headers-5.4.0-153 linux-headers-5.4.0-153-generic 2023-07-10T16:44:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: linux-headers-generic linux-image-5.4.0-153-generic 2023-07-10T16:44:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: linux-modules-5.4.0-153-generic linux-modules-extra-5.4.0-153-generic 2023-07-10T16:44:23.443 INFO:teuthology.orchestra.run.smithi192.stdout:Suggested packages: 2023-07-10T16:44:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-07-10T16:44:23.444 INFO:teuthology.orchestra.run.smithi192.stdout:Recommended packages: 2023-07-10T16:44:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: thermald 2023-07-10T16:44:23.488 INFO:teuthology.orchestra.run.smithi116.stdout:The following additional packages will be installed: 2023-07-10T16:44:23.488 INFO:teuthology.orchestra.run.smithi116.stdout: linux-generic linux-headers-5.4.0-153 linux-headers-5.4.0-153-generic 2023-07-10T16:44:23.488 INFO:teuthology.orchestra.run.smithi116.stdout: linux-headers-generic linux-image-5.4.0-153-generic 2023-07-10T16:44:23.489 INFO:teuthology.orchestra.run.smithi116.stdout: linux-modules-5.4.0-153-generic linux-modules-extra-5.4.0-153-generic 2023-07-10T16:44:23.489 INFO:teuthology.orchestra.run.smithi116.stdout:Suggested packages: 2023-07-10T16:44:23.489 INFO:teuthology.orchestra.run.smithi116.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-07-10T16:44:23.489 INFO:teuthology.orchestra.run.smithi116.stdout:Recommended packages: 2023-07-10T16:44:23.490 INFO:teuthology.orchestra.run.smithi116.stdout: thermald 2023-07-10T16:44:23.504 INFO:teuthology.orchestra.run.smithi192.stdout:The following NEW packages will be installed: 2023-07-10T16:44:23.504 INFO:teuthology.orchestra.run.smithi192.stdout: linux-headers-5.4.0-153 linux-headers-5.4.0-153-generic 2023-07-10T16:44:23.504 INFO:teuthology.orchestra.run.smithi192.stdout: linux-image-5.4.0-153-generic linux-modules-5.4.0-153-generic 2023-07-10T16:44:23.505 INFO:teuthology.orchestra.run.smithi192.stdout: linux-modules-extra-5.4.0-153-generic 2023-07-10T16:44:23.505 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages will be upgraded: 2023-07-10T16:44:23.505 INFO:teuthology.orchestra.run.smithi192.stdout: linux-generic linux-headers-generic linux-image-generic 2023-07-10T16:44:23.549 INFO:teuthology.orchestra.run.smithi116.stdout:The following NEW packages will be installed: 2023-07-10T16:44:23.549 INFO:teuthology.orchestra.run.smithi116.stdout: linux-headers-5.4.0-153 linux-headers-5.4.0-153-generic 2023-07-10T16:44:23.549 INFO:teuthology.orchestra.run.smithi116.stdout: linux-image-5.4.0-153-generic linux-modules-5.4.0-153-generic 2023-07-10T16:44:23.549 INFO:teuthology.orchestra.run.smithi116.stdout: linux-modules-extra-5.4.0-153-generic 2023-07-10T16:44:23.550 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages will be upgraded: 2023-07-10T16:44:23.550 INFO:teuthology.orchestra.run.smithi116.stdout: linux-generic linux-headers-generic linux-image-generic 2023-07-10T16:44:23.600 INFO:teuthology.orchestra.run.smithi116.stdout:3 upgraded, 5 newly installed, 0 to remove and 225 not upgraded. 2023-07-10T16:44:23.600 INFO:teuthology.orchestra.run.smithi116.stdout:Need to get 77.1 MB of archives. 2023-07-10T16:44:23.600 INFO:teuthology.orchestra.run.smithi116.stdout:After this operation, 380 MB of additional disk space will be used. 2023-07-10T16:44:23.601 INFO:teuthology.orchestra.run.smithi116.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-153-generic amd64 5.4.0-153.170 [15.0 MB] 2023-07-10T16:44:23.687 INFO:teuthology.orchestra.run.smithi192.stdout:3 upgraded, 5 newly installed, 0 to remove and 225 not upgraded. 2023-07-10T16:44:23.687 INFO:teuthology.orchestra.run.smithi192.stdout:Need to get 77.1 MB of archives. 2023-07-10T16:44:23.687 INFO:teuthology.orchestra.run.smithi192.stdout:After this operation, 380 MB of additional disk space will be used. 2023-07-10T16:44:23.687 INFO:teuthology.orchestra.run.smithi192.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-153-generic amd64 5.4.0-153.170 [15.0 MB] 2023-07-10T16:44:23.931 INFO:teuthology.orchestra.run.smithi116.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-153-generic amd64 5.4.0-153.170 [10.5 MB] 2023-07-10T16:44:24.036 INFO:teuthology.orchestra.run.smithi116.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-153-generic amd64 5.4.0-153.170 [39.2 MB] 2023-07-10T16:44:24.564 INFO:teuthology.orchestra.run.smithi116.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.153.150 [1,904 B] 2023-07-10T16:44:24.569 INFO:teuthology.orchestra.run.smithi116.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.153.150 [2,604 B] 2023-07-10T16:44:24.571 INFO:teuthology.orchestra.run.smithi116.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-153 all 5.4.0-153.170 [11.0 MB] 2023-07-10T16:44:24.669 INFO:teuthology.orchestra.run.smithi116.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-153-generic amd64 5.4.0-153.170 [1,363 kB] 2023-07-10T16:44:24.680 INFO:teuthology.orchestra.run.smithi116.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.153.150 [2,464 B] 2023-07-10T16:44:24.858 INFO:teuthology.orchestra.run.smithi192.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-153-generic amd64 5.4.0-153.170 [10.5 MB] 2023-07-10T16:44:25.128 INFO:teuthology.orchestra.run.smithi192.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-153-generic amd64 5.4.0-153.170 [39.2 MB] 2023-07-10T16:44:25.361 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 77.1 MB in 1s (68.6 MB/s) 2023-07-10T16:44:25.544 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-modules-5.4.0-153-generic. 2023-07-10T16:44:26.427 INFO:teuthology.orchestra.run.smithi192.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.153.150 [1,904 B] 2023-07-10T16:44:26.427 INFO:teuthology.orchestra.run.smithi192.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.153.150 [2,604 B] 2023-07-10T16:44:26.428 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-153 all 5.4.0-153.170 [11.0 MB] 2023-07-10T16:44:26.703 INFO:teuthology.orchestra.run.smithi192.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-153-generic amd64 5.4.0-153.170 [1,363 kB] 2023-07-10T16:44:26.779 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.153.150 [2,464 B] 2023-07-10T16:44:27.252 INFO:teuthology.orchestra.run.smithi116.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-07-10T16:44:27.257 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../0-linux-modules-5.4.0-153-generic_5.4.0-153.170_amd64.deb ... 2023-07-10T16:44:27.350 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-modules-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:27.508 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 77.1 MB in 3s (23.6 MB/s) 2023-07-10T16:44:27.708 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-modules-5.4.0-153-generic. 2023-07-10T16:44:29.191 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-image-5.4.0-153-generic. 2023-07-10T16:44:29.207 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../1-linux-image-5.4.0-153-generic_5.4.0-153.170_amd64.deb ... 2023-07-10T16:44:29.303 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-image-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:29.431 INFO:teuthology.orchestra.run.smithi192.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-07-10T16:44:29.436 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../0-linux-modules-5.4.0-153-generic_5.4.0-153.170_amd64.deb ... 2023-07-10T16:44:29.513 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-modules-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:30.394 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-153-generic. 2023-07-10T16:44:30.410 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-153-generic_5.4.0-153.170_amd64.deb ... 2023-07-10T16:44:30.452 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-modules-extra-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:31.479 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-image-5.4.0-153-generic. 2023-07-10T16:44:31.494 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../1-linux-image-5.4.0-153-generic_5.4.0-153.170_amd64.deb ... 2023-07-10T16:44:31.592 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-image-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:32.682 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-153-generic. 2023-07-10T16:44:32.698 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-153-generic_5.4.0-153.170_amd64.deb ... 2023-07-10T16:44:32.740 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-modules-extra-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:35.114 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../3-linux-generic_5.4.0.153.150_amd64.deb ... 2023-07-10T16:44:35.242 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-generic (5.4.0.153.150) over (5.4.0.124.125) ... 2023-07-10T16:44:35.602 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.153.150_amd64.deb ... 2023-07-10T16:44:35.728 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-image-generic (5.4.0.153.150) over (5.4.0.124.125) ... 2023-07-10T16:44:36.065 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-headers-5.4.0-153. 2023-07-10T16:44:36.079 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../5-linux-headers-5.4.0-153_5.4.0-153.170_all.deb ... 2023-07-10T16:44:36.123 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-headers-5.4.0-153 (5.4.0-153.170) ... 2023-07-10T16:44:37.508 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../3-linux-generic_5.4.0.153.150_amd64.deb ... 2023-07-10T16:44:37.635 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-generic (5.4.0.153.150) over (5.4.0.124.125) ... 2023-07-10T16:44:38.011 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.153.150_amd64.deb ... 2023-07-10T16:44:38.138 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-image-generic (5.4.0.153.150) over (5.4.0.124.125) ... 2023-07-10T16:44:38.466 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-headers-5.4.0-153. 2023-07-10T16:44:38.482 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../5-linux-headers-5.4.0-153_5.4.0-153.170_all.deb ... 2023-07-10T16:44:38.524 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-headers-5.4.0-153 (5.4.0-153.170) ... 2023-07-10T16:44:41.327 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-headers-5.4.0-153-generic. 2023-07-10T16:44:41.350 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../6-linux-headers-5.4.0-153-generic_5.4.0-153.170_amd64.deb ... 2023-07-10T16:44:41.393 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-headers-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:43.895 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-headers-5.4.0-153-generic. 2023-07-10T16:44:43.918 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../6-linux-headers-5.4.0-153-generic_5.4.0-153.170_amd64.deb ... 2023-07-10T16:44:43.962 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-headers-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:44.019 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.153.150_amd64.deb ... 2023-07-10T16:44:44.156 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-headers-generic (5.4.0.153.150) over (5.4.0.124.125) ... 2023-07-10T16:44:44.494 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-headers-5.4.0-153 (5.4.0-153.170) ... 2023-07-10T16:44:44.619 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-modules-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:45.838 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-image-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:46.631 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.153.150_amd64.deb ... 2023-07-10T16:44:46.767 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-headers-generic (5.4.0.153.150) over (5.4.0.124.125) ... 2023-07-10T16:44:47.004 INFO:teuthology.orchestra.run.smithi116.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-153-generic 2023-07-10T16:44:47.004 INFO:teuthology.orchestra.run.smithi116.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-153-generic 2023-07-10T16:44:47.120 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-headers-5.4.0-153 (5.4.0-153.170) ... 2023-07-10T16:44:47.148 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-headers-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:47.247 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-modules-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:47.274 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-headers-generic (5.4.0.153.150) ... 2023-07-10T16:44:47.401 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-modules-extra-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:48.458 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-image-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:48.585 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-image-generic (5.4.0.153.150) ... 2023-07-10T16:44:48.711 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-generic (5.4.0.153.150) ... 2023-07-10T16:44:48.862 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for linux-image-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:48.935 INFO:teuthology.orchestra.run.smithi116.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-07-10T16:44:48.936 INFO:teuthology.orchestra.run.smithi116.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-153-generic 2023-07-10T16:44:49.613 INFO:teuthology.orchestra.run.smithi192.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-153-generic 2023-07-10T16:44:49.614 INFO:teuthology.orchestra.run.smithi192.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-153-generic 2023-07-10T16:44:49.777 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-headers-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:49.903 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-headers-generic (5.4.0.153.150) ... 2023-07-10T16:44:50.029 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-modules-extra-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:51.239 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-image-generic (5.4.0.153.150) ... 2023-07-10T16:44:51.373 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-generic (5.4.0.153.150) ... 2023-07-10T16:44:51.500 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for linux-image-5.4.0-153-generic (5.4.0-153.170) ... 2023-07-10T16:44:51.575 INFO:teuthology.orchestra.run.smithi192.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-07-10T16:44:51.576 INFO:teuthology.orchestra.run.smithi192.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-153-generic 2023-07-10T16:45:11.308 INFO:teuthology.orchestra.run.smithi116.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-07-10T16:45:11.308 INFO:teuthology.orchestra.run.smithi116.stdout:Sourcing file `/etc/default/grub' 2023-07-10T16:45:11.316 INFO:teuthology.orchestra.run.smithi116.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-07-10T16:45:11.327 INFO:teuthology.orchestra.run.smithi116.stdout:Generating grub configuration file ... 2023-07-10T16:45:11.647 INFO:teuthology.orchestra.run.smithi116.stdout:Found linux image: /boot/vmlinuz-5.4.0-153-generic 2023-07-10T16:45:11.664 INFO:teuthology.orchestra.run.smithi116.stdout:Found initrd image: /boot/initrd.img-5.4.0-153-generic 2023-07-10T16:45:12.022 INFO:teuthology.orchestra.run.smithi116.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-07-10T16:45:12.033 INFO:teuthology.orchestra.run.smithi116.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-07-10T16:45:13.684 INFO:teuthology.orchestra.run.smithi192.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-07-10T16:45:13.684 INFO:teuthology.orchestra.run.smithi192.stdout:Sourcing file `/etc/default/grub' 2023-07-10T16:45:13.694 INFO:teuthology.orchestra.run.smithi192.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-07-10T16:45:13.713 INFO:teuthology.orchestra.run.smithi192.stdout:Generating grub configuration file ... 2023-07-10T16:45:13.999 INFO:teuthology.orchestra.run.smithi192.stdout:Found linux image: /boot/vmlinuz-5.4.0-153-generic 2023-07-10T16:45:14.019 INFO:teuthology.orchestra.run.smithi192.stdout:Found initrd image: /boot/initrd.img-5.4.0-153-generic 2023-07-10T16:45:14.037 INFO:teuthology.orchestra.run.smithi116.stdout:done 2023-07-10T16:45:14.426 INFO:teuthology.orchestra.run.smithi192.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-07-10T16:45:14.435 DEBUG:teuthology.orchestra.run.smithi116:> dpkg -s linux-image-generic 2023-07-10T16:45:14.437 INFO:teuthology.orchestra.run.smithi192.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-07-10T16:45:14.492 INFO:teuthology.orchestra.run.smithi116.stdout:Package: linux-image-generic 2023-07-10T16:45:14.492 INFO:teuthology.orchestra.run.smithi116.stdout:Status: install ok installed 2023-07-10T16:45:14.493 INFO:teuthology.orchestra.run.smithi116.stdout:Priority: optional 2023-07-10T16:45:14.493 INFO:teuthology.orchestra.run.smithi116.stdout:Section: kernel 2023-07-10T16:45:14.493 INFO:teuthology.orchestra.run.smithi116.stdout:Installed-Size: 19 2023-07-10T16:45:14.493 INFO:teuthology.orchestra.run.smithi116.stdout:Maintainer: Ubuntu Kernel Team 2023-07-10T16:45:14.493 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: amd64 2023-07-10T16:45:14.493 INFO:teuthology.orchestra.run.smithi116.stdout:Source: linux-meta 2023-07-10T16:45:14.493 INFO:teuthology.orchestra.run.smithi116.stdout:Version: 5.4.0.153.150 2023-07-10T16:45:14.494 INFO:teuthology.orchestra.run.smithi116.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-07-10T16:45:14.494 INFO:teuthology.orchestra.run.smithi116.stdout:Depends: linux-image-5.4.0-153-generic, linux-modules-extra-5.4.0-153-generic, linux-firmware, intel-microcode, amd64-microcode 2023-07-10T16:45:14.494 INFO:teuthology.orchestra.run.smithi116.stdout:Recommends: thermald 2023-07-10T16:45:14.494 INFO:teuthology.orchestra.run.smithi116.stdout:Description: Generic Linux kernel image 2023-07-10T16:45:14.494 INFO:teuthology.orchestra.run.smithi116.stdout: This package will always depend on the latest generic kernel image 2023-07-10T16:45:14.494 INFO:teuthology.orchestra.run.smithi116.stdout: available. 2023-07-10T16:45:14.495 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-153-generic 2023-07-10T16:45:14.495 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-07-10T16:45:14.495 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-07-10T16:45:14.495 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-07-10T16:45:14.495 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2023-07-10T16:45:14.496 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get clean 2023-07-10T16:45:14.577 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get update 2023-07-10T16:45:14.708 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-07-10T16:45:14.709 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-07-10T16:45:14.715 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-07-10T16:45:14.732 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-07-10T16:45:16.414 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2023-07-10T16:45:16.439 DEBUG:teuthology.orchestra.run.smithi116:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-07-10T16:45:16.505 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2023-07-10T16:45:16.545 INFO:teuthology.orchestra.run.smithi192.stdout:done 2023-07-10T16:45:16.700 INFO:teuthology.orchestra.run.smithi116.stdout:Building dependency tree... 2023-07-10T16:45:16.701 INFO:teuthology.orchestra.run.smithi116.stdout:Reading state information... 2023-07-10T16:45:16.914 INFO:teuthology.orchestra.run.smithi116.stdout:linux-image-generic is already the newest version (5.4.0.153.150). 2023-07-10T16:45:16.914 INFO:teuthology.orchestra.run.smithi116.stdout:0 upgraded, 0 newly installed, 0 to remove and 225 not upgraded. 2023-07-10T16:45:16.916 DEBUG:teuthology.orchestra.run.smithi116:> dpkg -s linux-image-generic 2023-07-10T16:45:16.932 INFO:teuthology.orchestra.run.smithi116.stdout:Package: linux-image-generic 2023-07-10T16:45:16.933 INFO:teuthology.orchestra.run.smithi116.stdout:Status: install ok installed 2023-07-10T16:45:16.933 INFO:teuthology.orchestra.run.smithi116.stdout:Priority: optional 2023-07-10T16:45:16.933 INFO:teuthology.orchestra.run.smithi116.stdout:Section: kernel 2023-07-10T16:45:16.933 INFO:teuthology.orchestra.run.smithi116.stdout:Installed-Size: 19 2023-07-10T16:45:16.933 INFO:teuthology.orchestra.run.smithi116.stdout:Maintainer: Ubuntu Kernel Team 2023-07-10T16:45:16.934 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: amd64 2023-07-10T16:45:16.934 INFO:teuthology.orchestra.run.smithi116.stdout:Source: linux-meta 2023-07-10T16:45:16.934 INFO:teuthology.orchestra.run.smithi116.stdout:Version: 5.4.0.153.150 2023-07-10T16:45:16.934 INFO:teuthology.orchestra.run.smithi116.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-07-10T16:45:16.934 INFO:teuthology.orchestra.run.smithi116.stdout:Depends: linux-image-5.4.0-153-generic, linux-modules-extra-5.4.0-153-generic, linux-firmware, intel-microcode, amd64-microcode 2023-07-10T16:45:16.934 INFO:teuthology.orchestra.run.smithi116.stdout:Recommends: thermald 2023-07-10T16:45:16.934 INFO:teuthology.orchestra.run.smithi116.stdout:Description: Generic Linux kernel image 2023-07-10T16:45:16.935 INFO:teuthology.orchestra.run.smithi116.stdout: This package will always depend on the latest generic kernel image 2023-07-10T16:45:16.935 INFO:teuthology.orchestra.run.smithi116.stdout: available. 2023-07-10T16:45:16.935 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2023-07-10T16:45:16.939 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.UNmRmxWZ2c 2023-07-10T16:45:16.940 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub/grub.cfg /tmp/tmp.UNmRmxWZ2c 2023-07-10T16:45:16.954 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.UNmRmxWZ2c 2023-07-10T16:45:16.956 DEBUG:teuthology.orchestra.run.smithi192:> dpkg -s linux-image-generic 2023-07-10T16:45:16.991 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.UNmRmxWZ2c is 9KB 2023-07-10T16:45:17.005 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.UNmRmxWZ2c 2023-07-10T16:45:17.008 INFO:teuthology.orchestra.run.smithi192.stdout:Package: linux-image-generic 2023-07-10T16:45:17.009 INFO:teuthology.orchestra.run.smithi192.stdout:Status: install ok installed 2023-07-10T16:45:17.009 INFO:teuthology.orchestra.run.smithi192.stdout:Priority: optional 2023-07-10T16:45:17.009 INFO:teuthology.orchestra.run.smithi192.stdout:Section: kernel 2023-07-10T16:45:17.009 INFO:teuthology.orchestra.run.smithi192.stdout:Installed-Size: 19 2023-07-10T16:45:17.009 INFO:teuthology.orchestra.run.smithi192.stdout:Maintainer: Ubuntu Kernel Team 2023-07-10T16:45:17.010 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: amd64 2023-07-10T16:45:17.010 INFO:teuthology.orchestra.run.smithi192.stdout:Source: linux-meta 2023-07-10T16:45:17.010 INFO:teuthology.orchestra.run.smithi192.stdout:Version: 5.4.0.153.150 2023-07-10T16:45:17.010 INFO:teuthology.orchestra.run.smithi192.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-07-10T16:45:17.010 INFO:teuthology.orchestra.run.smithi192.stdout:Depends: linux-image-5.4.0-153-generic, linux-modules-extra-5.4.0-153-generic, linux-firmware, intel-microcode, amd64-microcode 2023-07-10T16:45:17.010 INFO:teuthology.orchestra.run.smithi192.stdout:Recommends: thermald 2023-07-10T16:45:17.010 INFO:teuthology.orchestra.run.smithi192.stdout:Description: Generic Linux kernel image 2023-07-10T16:45:17.010 INFO:teuthology.orchestra.run.smithi192.stdout: This package will always depend on the latest generic kernel image 2023-07-10T16:45:17.011 INFO:teuthology.orchestra.run.smithi192.stdout: available. 2023-07-10T16:45:17.011 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-153-generic 2023-07-10T16:45:17.011 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-07-10T16:45:17.011 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-07-10T16:45:17.011 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-07-10T16:45:17.012 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi192.front.sepia.ceph.com, path=None, version=distro) 2023-07-10T16:45:17.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get clean 2023-07-10T16:45:17.013 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-07-10T16:45:17.068 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-07-10T16:45:17.069 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-07-10T16:45:17.069 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-07-10T16:45:17.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get update 2023-07-10T16:45:17.101 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-153-generic 2023-07-10T16:45:17.101 DEBUG:teuthology.orchestra.run.smithi116:> sudo update-grub 2023-07-10T16:45:17.225 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-07-10T16:45:17.341 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-07-10T16:45:17.423 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-07-10T16:45:17.507 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-07-10T16:45:18.355 INFO:teuthology.orchestra.run.smithi116.stderr:Sourcing file `/etc/default/grub' 2023-07-10T16:45:18.360 INFO:teuthology.orchestra.run.smithi116.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-07-10T16:45:18.375 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2023-07-10T16:45:18.621 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-5.4.0-153-generic 2023-07-10T16:45:18.632 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-5.4.0-153-generic 2023-07-10T16:45:18.929 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2023-07-10T16:45:18.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-07-10T16:45:18.979 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-07-10T16:45:18.991 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-07-10T16:45:19.019 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2023-07-10T16:45:19.236 INFO:teuthology.orchestra.run.smithi192.stdout:Building dependency tree... 2023-07-10T16:45:19.237 INFO:teuthology.orchestra.run.smithi192.stdout:Reading state information... 2023-07-10T16:45:19.457 INFO:teuthology.orchestra.run.smithi192.stdout:linux-image-generic is already the newest version (5.4.0.153.150). 2023-07-10T16:45:19.457 INFO:teuthology.orchestra.run.smithi192.stdout:0 upgraded, 0 newly installed, 0 to remove and 225 not upgraded. 2023-07-10T16:45:19.458 DEBUG:teuthology.orchestra.run.smithi192:> dpkg -s linux-image-generic 2023-07-10T16:45:19.478 INFO:teuthology.orchestra.run.smithi192.stdout:Package: linux-image-generic 2023-07-10T16:45:19.478 INFO:teuthology.orchestra.run.smithi192.stdout:Status: install ok installed 2023-07-10T16:45:19.479 INFO:teuthology.orchestra.run.smithi192.stdout:Priority: optional 2023-07-10T16:45:19.479 INFO:teuthology.orchestra.run.smithi192.stdout:Section: kernel 2023-07-10T16:45:19.479 INFO:teuthology.orchestra.run.smithi192.stdout:Installed-Size: 19 2023-07-10T16:45:19.479 INFO:teuthology.orchestra.run.smithi192.stdout:Maintainer: Ubuntu Kernel Team 2023-07-10T16:45:19.479 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: amd64 2023-07-10T16:45:19.479 INFO:teuthology.orchestra.run.smithi192.stdout:Source: linux-meta 2023-07-10T16:45:19.479 INFO:teuthology.orchestra.run.smithi192.stdout:Version: 5.4.0.153.150 2023-07-10T16:45:19.480 INFO:teuthology.orchestra.run.smithi192.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-07-10T16:45:19.480 INFO:teuthology.orchestra.run.smithi192.stdout:Depends: linux-image-5.4.0-153-generic, linux-modules-extra-5.4.0-153-generic, linux-firmware, intel-microcode, amd64-microcode 2023-07-10T16:45:19.480 INFO:teuthology.orchestra.run.smithi192.stdout:Recommends: thermald 2023-07-10T16:45:19.480 INFO:teuthology.orchestra.run.smithi192.stdout:Description: Generic Linux kernel image 2023-07-10T16:45:19.480 INFO:teuthology.orchestra.run.smithi192.stdout: This package will always depend on the latest generic kernel image 2023-07-10T16:45:19.480 INFO:teuthology.orchestra.run.smithi192.stdout: available. 2023-07-10T16:45:19.481 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2023-07-10T16:45:19.485 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.b9EGFHENJH 2023-07-10T16:45:19.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo cp /boot/grub/grub.cfg /tmp/tmp.b9EGFHENJH 2023-07-10T16:45:19.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.b9EGFHENJH 2023-07-10T16:45:19.629 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.b9EGFHENJH is 9KB 2023-07-10T16:45:19.644 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.b9EGFHENJH 2023-07-10T16:45:19.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-07-10T16:45:19.708 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-10T16:45:19.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-07-10T16:45:19.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-07-10T16:45:19.746 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-153-generic 2023-07-10T16:45:19.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo update-grub 2023-07-10T16:45:19.962 INFO:teuthology.orchestra.run.smithi116.stderr:done 2023-07-10T16:45:19.964 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2023-07-10T16:45:21.081 INFO:teuthology.orchestra.run.smithi192.stderr:Sourcing file `/etc/default/grub' 2023-07-10T16:45:21.088 INFO:teuthology.orchestra.run.smithi192.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-07-10T16:45:21.096 INFO:teuthology.orchestra.run.smithi192.stderr:Generating grub configuration file ... 2023-07-10T16:45:21.399 INFO:teuthology.orchestra.run.smithi192.stderr:Found linux image: /boot/vmlinuz-5.4.0-153-generic 2023-07-10T16:45:21.411 INFO:teuthology.orchestra.run.smithi192.stderr:Found initrd image: /boot/initrd.img-5.4.0-153-generic 2023-07-10T16:45:21.806 INFO:teuthology.orchestra.run.smithi192.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-07-10T16:45:21.818 INFO:teuthology.orchestra.run.smithi192.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-07-10T16:45:22.788 INFO:teuthology.orchestra.run.smithi192.stderr:done 2023-07-10T16:45:22.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo shutdown -r now 2023-07-10T16:45:49.993 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-07-10T16:45:49.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-07-10T16:45:49.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-10T16:45:52.792 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-07-10T16:45:52.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-07-10T16:45:52.793 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-10T16:46:27.444 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-07-10T16:46:36.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-07-10T16:46:36.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-10T16:46:50.008 DEBUG:teuthology.orchestra.remote:timed out 2023-07-10T16:46:54.832 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-07-10T16:46:59.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-07-10T16:46:59.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-10T16:46:59.365 DEBUG:teuthology.orchestra.run.smithi116:> true 2023-07-10T16:47:00.204 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-07-10T16:47:00.205 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-153-generic"... 2023-07-10T16:47:00.205 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2023-07-10T16:47:00.252 INFO:teuthology.orchestra.run.smithi116.stdout:5.4.0-153-generic 2023-07-10T16:47:00.253 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-153-generic vs 5.4.0-153-generic 2023-07-10T16:47:00.253 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-07-10T16:47:00.253 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-07-10T16:47:01.254 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-07-10T16:47:01.255 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-10T16:47:01.388 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2023-07-10T16:47:01.421 DEBUG:teuthology.parallel:result is None 2023-07-10T16:47:06.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-07-10T16:47:06.835 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-10T16:47:07.034 DEBUG:teuthology.orchestra.run.smithi192:> true 2023-07-10T16:47:07.659 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-07-10T16:47:07.659 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-153-generic"... 2023-07-10T16:47:07.659 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2023-07-10T16:47:07.666 INFO:teuthology.orchestra.run.smithi192.stdout:5.4.0-153-generic 2023-07-10T16:47:07.666 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-153-generic vs 5.4.0-153-generic 2023-07-10T16:47:07.666 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-07-10T16:47:07.666 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-07-10T16:47:08.667 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-07-10T16:47:08.668 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-10T16:47:08.765 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2023-07-10T16:47:08.792 DEBUG:teuthology.parallel:result is None 2023-07-10T16:47:08.792 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-10T16:47:08.800 INFO:teuthology.task.internal:Creating test directory... 2023-07-10T16:47:08.801 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-10T16:47:08.804 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-10T16:47:08.811 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-10T16:47:08.846 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-10T16:47:08.875 INFO:teuthology.task.internal:Creating archive directory... 2023-07-10T16:47:08.876 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-10T16:47:08.879 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-10T16:47:08.909 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-10T16:47:08.922 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-10T16:47:08.923 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-10T16:47:08.933 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-10T16:47:08.954 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-10T16:47:08.958 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-10T16:47:08.965 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-10T16:47:08.968 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-10T16:47:08.970 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-10T16:47:08.982 INFO:teuthology.task.internal:Configuring sudo... 2023-07-10T16:47:08.982 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-10T16:47:09.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-10T16:47:09.029 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-10T16:47:09.038 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-10T16:47:09.039 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-10T16:47:09.041 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-10T16:47:09.076 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-10T16:47:09.093 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-10T16:47:09.100 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-07-10T16:47:09.100 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-10T16:47:09.124 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-10T16:47:09.131 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-10T16:47:09.138 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-10T16:47:09.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-10T16:47:09.161 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2023-07-10T16:47:09.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2023-07-10T16:47:09.257 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-10T16:47:09.269 INFO:teuthology.task.internal:Starting timer... 2023-07-10T16:47:09.270 INFO:teuthology.run_tasks:Running task pcp... 2023-07-10T16:47:09.300 INFO:teuthology.run_tasks:Running task selinux... 2023-07-10T16:47:09.329 DEBUG:teuthology.task.selinux:Excluding smithi116: OS 'ubuntu' does not support SELinux 2023-07-10T16:47:09.330 DEBUG:teuthology.task.selinux:Excluding smithi192: OS 'ubuntu' does not support SELinux 2023-07-10T16:47:09.330 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-10T16:47:09.331 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-07-10T16:47:09.331 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-10T16:47:09.331 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-10T16:47:09.364 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-07-10T16:47:09.371 INFO:teuthology.repo_utils:Fetching main from origin 2023-07-10T16:47:09.512 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-10T16:47:09.536 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-10T16:47:09.537 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi116.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-10T16:53:00.318 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2023-07-10T16:53:00.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-07-10T16:53:00.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-10T16:53:00.367 DEBUG:teuthology.orchestra.run.smithi116:> true 2023-07-10T16:53:00.485 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-07-10T16:53:00.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-07-10T16:53:00.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-10T16:53:00.535 DEBUG:teuthology.orchestra.run.smithi192:> true 2023-07-10T16:53:00.665 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-07-10T16:53:00.666 INFO:teuthology.run_tasks:Running task clock... 2023-07-10T16:53:00.678 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-10T16:53:00.679 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-10T16:53:00.680 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-10T16:53:00.682 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-10T16:53:00.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-10T16:53:00.714 INFO:teuthology.orchestra.run.smithi192.stdout:10 Jul 16:53:00 ntpd[22237]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-07-10T16:53:00.714 INFO:teuthology.orchestra.run.smithi192.stdout:10 Jul 16:53:00 ntpd[22237]: Command line: ntpd -gq 2023-07-10T16:53:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:10 Jul 16:53:00 ntpd[22267]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-07-10T16:53:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:10 Jul 16:53:00 ntpd[22267]: Command line: ntpd -gq 2023-07-10T16:53:00.715 INFO:teuthology.orchestra.run.smithi192.stdout:10 Jul 16:53:00 ntpd[22237]: proto: precision = 0.116 usec (-23) 2023-07-10T16:53:00.716 INFO:teuthology.orchestra.run.smithi192.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-07-10T16:53:00.716 INFO:teuthology.orchestra.run.smithi192.stdout:10 Jul 16:53:00 ntpd[22237]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-07-10T16:53:00.716 INFO:teuthology.orchestra.run.smithi192.stderr:restrict ::: KOD does nothing without LIMITED. 2023-07-10T16:53:00.717 INFO:teuthology.orchestra.run.smithi192.stdout:10 Jul 16:53:00 ntpd[22237]: restrict ::: KOD does nothing without LIMITED. 2023-07-10T16:53:00.717 INFO:teuthology.orchestra.run.smithi192.stdout:10 Jul 16:53:00 ntpd[22237]: Listen and drop on 0 v6wildcard [::]:123 2023-07-10T16:53:00.717 INFO:teuthology.orchestra.run.smithi192.stdout:10 Jul 16:53:00 ntpd[22237]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-07-10T16:53:00.718 INFO:teuthology.orchestra.run.smithi192.stdout:10 Jul 16:53:00 ntpd[22237]: Listen normally on 2 lo 127.0.0.1:123 2023-07-10T16:53:00.718 INFO:teuthology.orchestra.run.smithi192.stdout:10 Jul 16:53:00 ntpd[22237]: Listen normally on 3 enp3s0f1 172.21.15.192:123 2023-07-10T16:53:00.718 INFO:teuthology.orchestra.run.smithi192.stdout:10 Jul 16:53:00 ntpd[22237]: Listen normally on 4 lo [::1]:123 2023-07-10T16:53:00.719 INFO:teuthology.orchestra.run.smithi192.stdout:10 Jul 16:53:00 ntpd[22237]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8049%5]:123 2023-07-10T16:53:00.719 INFO:teuthology.orchestra.run.smithi192.stdout:10 Jul 16:53:00 ntpd[22237]: Listening on routing socket on fd #22 for interface updates 2023-07-10T16:53:00.719 INFO:teuthology.orchestra.run.smithi116.stdout:10 Jul 16:53:00 ntpd[22267]: proto: precision = 0.103 usec (-23) 2023-07-10T16:53:00.720 INFO:teuthology.orchestra.run.smithi116.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-07-10T16:53:00.721 INFO:teuthology.orchestra.run.smithi116.stdout:10 Jul 16:53:00 ntpd[22267]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-07-10T16:53:00.721 INFO:teuthology.orchestra.run.smithi116.stderr:restrict ::: KOD does nothing without LIMITED. 2023-07-10T16:53:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:10 Jul 16:53:00 ntpd[22267]: restrict ::: KOD does nothing without LIMITED. 2023-07-10T16:53:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:10 Jul 16:53:00 ntpd[22267]: Listen and drop on 0 v6wildcard [::]:123 2023-07-10T16:53:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:10 Jul 16:53:00 ntpd[22267]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-07-10T16:53:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:10 Jul 16:53:00 ntpd[22267]: Listen normally on 2 lo 127.0.0.1:123 2023-07-10T16:53:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:10 Jul 16:53:00 ntpd[22267]: Listen normally on 3 enp3s0f1 172.21.15.116:123 2023-07-10T16:53:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:10 Jul 16:53:00 ntpd[22267]: Listen normally on 4 lo [::1]:123 2023-07-10T16:53:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:10 Jul 16:53:00 ntpd[22267]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:3f67%5]:123 2023-07-10T16:53:00.724 INFO:teuthology.orchestra.run.smithi116.stdout:10 Jul 16:53:00 ntpd[22267]: Listening on routing socket on fd #22 for interface updates 2023-07-10T16:53:02.715 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:02 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:02.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:02 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:02.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:02 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:02.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:02 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:02.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:02 ntpd[22237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:02.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:02 ntpd[22237]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:02 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:02 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:02 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:02 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:02 ntpd[22267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:02 ntpd[22267]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:03.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:03 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:03.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:03 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:03.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:03 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:03.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:03 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:03.718 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:03 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:03.718 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:03 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:03.718 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:03 ntpd[22237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:03.719 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:03 ntpd[22237]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:03.719 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:03 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:03.719 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:03 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:03.719 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:03 ntpd[22237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:03.720 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:03 ntpd[22237]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:03.721 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:03 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:03.722 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:03 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:03.722 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:03 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:03.722 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:03 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:03.722 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:03 ntpd[22267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:03.723 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:03 ntpd[22267]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:04.715 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:04 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:04.715 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:04 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:04.715 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:04 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:04.715 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:04 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:04.715 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:04 ntpd[22237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:04.715 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:04 ntpd[22237]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:04 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:04 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:04 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:04 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:04 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:04 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:04 ntpd[22267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:04 ntpd[22267]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:05.715 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:05 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:05.715 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:05 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:05.715 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:05 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:05.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:05 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:05.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:05 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:05.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:05 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:05.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:05 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:05.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:05 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:05.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:05 ntpd[22237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:05.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:05 ntpd[22237]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:05.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:05 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:05.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:05 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:05.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:05 ntpd[22237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:05.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:05 ntpd[22237]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:05 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:05 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:05 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:05 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:05 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:05 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:05 ntpd[22267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:05 ntpd[22267]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:05 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:05 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:05 ntpd[22267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:05 ntpd[22267]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:06.715 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:06 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:06.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:06 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:06.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:06 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:06.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:06 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:06.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:06 ntpd[22237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:06.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:06 ntpd[22237]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:06.718 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:06 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:06.718 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:06 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:06.718 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:06 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:06.718 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:06 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:06.719 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:06 ntpd[22267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:06.719 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:06 ntpd[22267]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:07.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:07 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:07.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:07 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:07.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:07 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:07.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:07 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:07.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:07 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:07.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:07 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:07.718 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:07 ntpd[22237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:07.718 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:07 ntpd[22237]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:07.718 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:07 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:07.718 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:07 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:07.718 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:07 ntpd[22237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:07.718 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:07 ntpd[22237]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:07.719 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:07 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:07.720 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:07 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:07.720 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:07 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:07.720 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:07 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:07.720 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:07 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:07.720 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:07 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:07 ntpd[22267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:07 ntpd[22267]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:07 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:07 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:07 ntpd[22267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:07 ntpd[22267]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:08.715 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:08 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:08.715 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:08 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:08.715 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:08 ntpd[22237]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:08.715 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:08 ntpd[22237]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:08.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:08 ntpd[22237]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:08.716 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:08 ntpd[22237]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:08.716 INFO:teuthology.orchestra.run.smithi192.stdout:10 Jul 16:53:08 ntpd[22237]: ntpd: time slew +0.005955 s 2023-07-10T16:53:08.716 INFO:teuthology.orchestra.run.smithi192.stdout:ntpd: time slew +0.005955s 2023-07-10T16:53:08.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:08 ntpd[22237]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-07-10T16:53:08.717 INFO:teuthology.orchestra.run.smithi192.stderr:10 Jul 16:53:08 ntpd[22237]: can't open /var/log/ntpstats/loopstats.20230710: Permission denied 2023-07-10T16:53:08.718 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:08 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:08.718 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:08 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:08.718 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:08 ntpd[22267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-07-10T16:53:08.718 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:08 ntpd[22267]: can't open /var/log/ntpstats/rawstats.20230710: Permission denied 2023-07-10T16:53:08.718 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:08 ntpd[22267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-07-10T16:53:08.719 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:08 ntpd[22267]: can't open /var/log/ntpstats/peerstats.20230710: Permission denied 2023-07-10T16:53:08.719 INFO:teuthology.orchestra.run.smithi116.stdout:10 Jul 16:53:08 ntpd[22267]: ntpd: time slew +0.001595 s 2023-07-10T16:53:08.719 INFO:teuthology.orchestra.run.smithi116.stdout:ntpd: time slew +0.001595s 2023-07-10T16:53:08.719 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:08 ntpd[22267]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-07-10T16:53:08.720 INFO:teuthology.orchestra.run.smithi116.stderr:10 Jul 16:53:08 ntpd[22267]: can't open /var/log/ntpstats/loopstats.20230710: Permission denied 2023-07-10T16:53:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: remote refid st t when poll reach delay offset jitter 2023-07-10T16:53:08.776 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================== 2023-07-10T16:53:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-07-10T16:53:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-07-10T16:53:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-07-10T16:53:08.777 INFO:teuthology.orchestra.run.smithi116.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-07-10T16:53:08.780 INFO:teuthology.orchestra.run.smithi192.stdout: remote refid st t when poll reach delay offset jitter 2023-07-10T16:53:08.780 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================== 2023-07-10T16:53:08.780 INFO:teuthology.orchestra.run.smithi192.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-07-10T16:53:08.781 INFO:teuthology.orchestra.run.smithi192.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-07-10T16:53:08.781 INFO:teuthology.orchestra.run.smithi192.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-07-10T16:53:08.781 INFO:teuthology.orchestra.run.smithi192.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-07-10T16:53:08.782 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-10T16:53:08.883 INFO:tasks.cephadm:Config: {'allow_ptrace': False, 'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v16.2.0', 'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '8507faa7ce284e46218d481e3ee9c411ac3dc32c'} 2023-07-10T16:53:08.884 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v16.2.0 2023-07-10T16:53:08.896 INFO:tasks.cephadm:Cluster fsid is 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:53:08.898 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-10T16:53:08.899 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.116', 'mon.c': '[v2:172.21.15.116:3301,v1:172.21.15.116:6790]', 'mon.b': '172.21.15.192'} 2023-07-10T16:53:08.900 INFO:tasks.cephadm:First mon is mon.a on smithi116 2023-07-10T16:53:08.900 INFO:tasks.cephadm:First mgr is y 2023-07-10T16:53:08.901 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-10T16:53:08.902 DEBUG:teuthology.orchestra.run.smithi116:> sudo hostname $(hostname -s) 2023-07-10T16:53:08.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo hostname $(hostname -s) 2023-07-10T16:53:08.938 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.0)... 2023-07-10T16:53:08.939 DEBUG:teuthology.orchestra.run.smithi116:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-10T16:53:09.182 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-rw-r-- 1 ubuntu ubuntu 288386 Jul 10 16:53 /home/ubuntu/cephtest/cephadm 2023-07-10T16:53:09.182 DEBUG:teuthology.orchestra.run.smithi192:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-10T16:53:09.360 INFO:teuthology.orchestra.run.smithi192.stdout:-rw-rw-r-- 1 ubuntu ubuntu 288386 Jul 10 16:53 /home/ubuntu/cephtest/cephadm 2023-07-10T16:53:09.361 DEBUG:teuthology.orchestra.run.smithi116:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-10T16:53:09.370 DEBUG:teuthology.orchestra.run.smithi192:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-10T16:53:09.397 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v16.2.0 on all hosts... 2023-07-10T16:53:09.398 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 pull 2023-07-10T16:53:09.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 pull 2023-07-10T16:53:09.688 INFO:teuthology.orchestra.run.smithi116.stderr:Pulling container image quay.io/ceph/ceph:v16.2.0... 2023-07-10T16:53:09.697 INFO:teuthology.orchestra.run.smithi192.stderr:Pulling container image quay.io/ceph/ceph:v16.2.0... 2023-07-10T16:54:02.043 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T16:54:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)", 2023-07-10T16:54:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "image_id": "24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15", 2023-07-10T16:54:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "repo_digests": [ 2023-07-10T16:54:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "quay.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a" 2023-07-10T16:54:02.045 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2023-07-10T16:54:02.045 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T16:54:07.682 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2023-07-10T16:54:07.682 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)", 2023-07-10T16:54:07.682 INFO:teuthology.orchestra.run.smithi192.stdout: "image_id": "24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15", 2023-07-10T16:54:07.682 INFO:teuthology.orchestra.run.smithi192.stdout: "repo_digests": [ 2023-07-10T16:54:07.682 INFO:teuthology.orchestra.run.smithi192.stdout: "quay.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a" 2023-07-10T16:54:07.683 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2023-07-10T16:54:07.683 INFO:teuthology.orchestra.run.smithi192.stdout:} 2023-07-10T16:54:07.705 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph 2023-07-10T16:54:07.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph 2023-07-10T16:54:07.740 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /etc/ceph 2023-07-10T16:54:07.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /etc/ceph 2023-07-10T16:54:07.794 INFO:tasks.cephadm:Writing seed config... 2023-07-10T16:54:07.795 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2023-07-10T16:54:07.795 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-10T16:54:07.796 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-10T16:54:07.796 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2023-07-10T16:54:07.796 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-10T16:54:07.796 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-10T16:54:07.796 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-10T16:54:07.797 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-10T16:54:07.797 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-10T16:54:07.798 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-07-10T16:54:07.798 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-10T16:54:07.803 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3f83c2ec-1f42-11ee-9b31-001a4aab830c mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2023-07-10T16:54:07.804 DEBUG:teuthology.orchestra.run.smithi116:mon.a> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.a.service 2023-07-10T16:54:07.806 DEBUG:teuthology.orchestra.run.smithi116:mgr.y> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.y.service 2023-07-10T16:54:07.853 INFO:tasks.cephadm:Bootstrapping... 2023-07-10T16:54:07.853 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 -v bootstrap --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.116 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-10T16:54:08.007 INFO:teuthology.orchestra.run.smithi116.stderr:Could not locate podman: podman not found 2023-07-10T16:54:08.008 INFO:teuthology.orchestra.run.smithi116.stderr:Verifying podman|docker is present... 2023-07-10T16:54:08.008 INFO:teuthology.orchestra.run.smithi116.stderr:Verifying lvm2 is present... 2023-07-10T16:54:08.008 INFO:teuthology.orchestra.run.smithi116.stderr:Verifying time synchronization is in place... 2023-07-10T16:54:08.009 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-enabled chrony.service 2023-07-10T16:54:08.015 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-07-10T16:54:08.015 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-active chrony.service 2023-07-10T16:54:08.016 INFO:journalctl@ceph.mon.a.smithi116.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T16:54:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: inactive 2023-07-10T16:54:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-enabled chronyd.service 2023-07-10T16:54:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2023-07-10T16:54:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-active chronyd.service 2023-07-10T16:54:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: inactive 2023-07-10T16:54:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-enabled systemd-timesyncd.service 2023-07-10T16:54:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: masked 2023-07-10T16:54:08.040 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-active systemd-timesyncd.service 2023-07-10T16:54:08.046 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: inactive 2023-07-10T16:54:08.047 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-enabled ntpd.service 2023-07-10T16:54:08.053 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2023-07-10T16:54:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-active ntpd.service 2023-07-10T16:54:08.059 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: inactive 2023-07-10T16:54:08.060 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-enabled ntp.service 2023-07-10T16:54:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: enabled 2023-07-10T16:54:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-active ntp.service 2023-07-10T16:54:08.074 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: active 2023-07-10T16:54:08.076 INFO:teuthology.orchestra.run.smithi116.stderr:Unit ntp.service is enabled and running 2023-07-10T16:54:08.076 INFO:teuthology.orchestra.run.smithi116.stderr:Repeating the final host check... 2023-07-10T16:54:08.077 INFO:teuthology.orchestra.run.smithi116.stderr:podman|docker (/usr/bin/docker) is present 2023-07-10T16:54:08.077 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl is present 2023-07-10T16:54:08.077 INFO:teuthology.orchestra.run.smithi116.stderr:lvcreate is present 2023-07-10T16:54:08.078 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-enabled chrony.service 2023-07-10T16:54:08.082 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-07-10T16:54:08.082 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-active chrony.service 2023-07-10T16:54:08.088 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: inactive 2023-07-10T16:54:08.089 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-enabled chronyd.service 2023-07-10T16:54:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2023-07-10T16:54:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-active chronyd.service 2023-07-10T16:54:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: inactive 2023-07-10T16:54:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-enabled systemd-timesyncd.service 2023-07-10T16:54:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: masked 2023-07-10T16:54:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-active systemd-timesyncd.service 2023-07-10T16:54:08.114 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: inactive 2023-07-10T16:54:08.115 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-enabled ntpd.service 2023-07-10T16:54:08.121 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2023-07-10T16:54:08.123 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-active ntpd.service 2023-07-10T16:54:08.124 INFO:journalctl@ceph.mgr.y.smithi116.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T16:54:08.128 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: inactive 2023-07-10T16:54:08.129 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-enabled ntp.service 2023-07-10T16:54:08.136 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: enabled 2023-07-10T16:54:08.137 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl is-active ntp.service 2023-07-10T16:54:08.144 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: active 2023-07-10T16:54:08.146 INFO:teuthology.orchestra.run.smithi116.stderr:Unit ntp.service is enabled and running 2023-07-10T16:54:08.147 INFO:teuthology.orchestra.run.smithi116.stderr:Host looks OK 2023-07-10T16:54:08.147 INFO:teuthology.orchestra.run.smithi116.stderr:Cluster fsid: 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:54:08.147 INFO:teuthology.orchestra.run.smithi116.stderr:Acquiring lock 140519027428464 on /run/cephadm/3f83c2ec-1f42-11ee-9b31-001a4aab830c.lock 2023-07-10T16:54:08.147 INFO:teuthology.orchestra.run.smithi116.stderr:Lock 140519027428464 acquired on /run/cephadm/3f83c2ec-1f42-11ee-9b31-001a4aab830c.lock 2023-07-10T16:54:08.148 INFO:teuthology.orchestra.run.smithi116.stderr:Verifying IP 172.21.15.116 port 3300 ... 2023-07-10T16:54:08.148 INFO:teuthology.orchestra.run.smithi116.stderr:Verifying IP 172.21.15.116 port 6789 ... 2023-07-10T16:54:08.148 INFO:teuthology.orchestra.run.smithi116.stderr:Base mon IP is 172.21.15.116, final addrv is [v2:172.21.15.116:3300,v1:172.21.15.116:6789] 2023-07-10T16:54:08.148 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/sbin/ip route ls 2023-07-10T16:54:08.150 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/sbin/ip: default via 172.21.15.254 dev enp3s0f1 2023-07-10T16:54:08.150 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.116 metric 100 2023-07-10T16:54:08.151 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/sbin/ip: 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2023-07-10T16:54:08.151 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.116 2023-07-10T16:54:08.151 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/sbin/ip: 172.21.15.254 dev enp3s0f1 proto dhcp scope link src 172.21.15.116 metric 100 2023-07-10T16:54:08.151 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/sbin/ip -6 route ls 2023-07-10T16:54:08.154 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-07-10T16:54:08.155 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2023-07-10T16:54:08.155 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 expires 1794sec pref medium 2023-07-10T16:54:08.155 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/sbin/ip -6 addr ls 2023-07-10T16:54:08.158 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-10T16:54:08.158 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/sbin/ip: inet6 ::1/128 scope host 2023-07-10T16:54:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/sbin/ip: valid_lft forever preferred_lft forever 2023-07-10T16:54:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-07-10T16:54:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/sbin/ip: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2023-07-10T16:54:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/sbin/ip: valid_lft forever preferred_lft forever 2023-07-10T16:54:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:Mon IP 172.21.15.116 is in CIDR network 172.21.0.0/20 2023-07-10T16:54:08.160 INFO:teuthology.orchestra.run.smithi116.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-10T16:54:08.161 INFO:teuthology.orchestra.run.smithi116.stderr:Pulling container image quay.io/ceph/ceph:v16.2.0... 2023-07-10T16:54:08.161 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker pull quay.io/ceph/ceph:v16.2.0 2023-07-10T16:54:08.572 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/docker: v16.2.0: Pulling from ceph/ceph 2023-07-10T16:54:08.677 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/docker: Digest: sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a 2023-07-10T16:54:08.678 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/docker: Status: Image is up to date for quay.io/ceph/ceph:v16.2.0 2023-07-10T16:54:08.702 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/docker: quay.io/ceph/ceph:v16.2.0 2023-07-10T16:54:08.705 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 --version 2023-07-10T16:54:09.810 INFO:teuthology.orchestra.run.smithi116.stderr:ceph: ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable) 2023-07-10T16:54:10.405 INFO:teuthology.orchestra.run.smithi116.stderr:Ceph version: ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable) 2023-07-10T16:54:10.406 INFO:teuthology.orchestra.run.smithi116.stderr:Extracting ceph user uid/gid from container image... 2023-07-10T16:54:10.406 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 -c %u %g /var/lib/ceph 2023-07-10T16:54:11.218 INFO:teuthology.orchestra.run.smithi116.stderr:stat: 167 167 2023-07-10T16:54:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:Creating initial keys... 2023-07-10T16:54:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 --gen-print-key 2023-07-10T16:54:12.714 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-authtool: AQC0N6xkb6w2KhAAVIlFuneDZxJccdUzlNya0w== 2023-07-10T16:54:13.315 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 --gen-print-key 2023-07-10T16:54:14.191 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-authtool: AQC2N6xkpOEICxAA0NOyKQ+Ppy9+J6UPhxwQ0Q== 2023-07-10T16:54:14.829 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 --gen-print-key 2023-07-10T16:54:15.680 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-authtool: AQC3N6xk9xkvKBAA1M3fpXdXODX3bpyGVYv5ew== 2023-07-10T16:54:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:Creating initial monmap... 2023-07-10T16:54:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmp816ygawe:/tmp/monmap:z quay.io/ceph/ceph:v16.2.0 --create --clobber --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c --addv a [v2:172.21.15.116:3300,v1:172.21.15.116:6789] /tmp/monmap 2023-07-10T16:54:17.180 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-10T16:54:17.181 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:54:17.181 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-10T16:54:17.780 INFO:teuthology.orchestra.run.smithi116.stderr:monmaptool for a [v2:172.21.15.116:3300,v1:172.21.15.116:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-10T16:54:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/monmaptool: set fsid to 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:54:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-10T16:54:17.781 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-07-10T16:54:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:Creating mon... 2023-07-10T16:54:17.784 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmpmp2atj2_:/tmp/keyring:z -v /tmp/ceph-tmp816ygawe:/tmp/monmap:z quay.io/ceph/ceph:v16.2.0 --mkfs -i a --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-07-10T16:54:18.665 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.655+0000 7fbae8dca700 0 set uid:gid to 167:167 (ceph:ceph) 2023-07-10T16:54:18.665 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 1 imported monmap: 2023-07-10T16:54:18.666 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: epoch 0 2023-07-10T16:54:18.666 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:54:18.666 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: last_changed 2023-07-10T16:54:17.174185+0000 2023-07-10T16:54:18.666 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: created 2023-07-10T16:54:17.174185+0000 2023-07-10T16:54:18.667 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-07-10T16:54:18.667 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-07-10T16:54:18.667 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-07-10T16:54:18.668 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.668 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 0 /usr/bin/ceph-mon: set fsid to 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:54:18.668 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: RocksDB version: 6.8.1 2023-07-10T16:54:18.668 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.669 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-07-10T16:54:18.669 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Compile date Mar 30 2021 2023-07-10T16:54:18.669 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: DB SUMMARY 2023-07-10T16:54:18.670 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.670 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2023-07-10T16:54:18.670 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.670 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2023-07-10T16:54:18.671 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.671 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.error_if_exists: 0 2023-07-10T16:54:18.671 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.create_if_missing: 1 2023-07-10T16:54:18.671 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.paranoid_checks: 1 2023-07-10T16:54:18.671 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.env: 0x5599c46321c0 2023-07-10T16:54:18.691 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.fs: Posix File System 2023-07-10T16:54:18.692 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.info_log: 0x5599c5f38f40 2023-07-10T16:54:18.692 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.max_file_opening_threads: 16 2023-07-10T16:54:18.692 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.statistics: (nil) 2023-07-10T16:54:18.692 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.use_fsync: 0 2023-07-10T16:54:18.693 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.max_log_file_size: 0 2023-07-10T16:54:18.693 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-10T16:54:18.693 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-07-10T16:54:18.693 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.keep_log_file_num: 1000 2023-07-10T16:54:18.693 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.recycle_log_file_num: 0 2023-07-10T16:54:18.693 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.allow_fallocate: 1 2023-07-10T16:54:18.693 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.allow_mmap_reads: 0 2023-07-10T16:54:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.allow_mmap_writes: 0 2023-07-10T16:54:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.use_direct_reads: 0 2023-07-10T16:54:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-10T16:54:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.create_missing_column_families: 0 2023-07-10T16:54:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.db_log_dir: 2023-07-10T16:54:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2023-07-10T16:54:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-07-10T16:54:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.max_subcompactions: 1 2023-07-10T16:54:18.695 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.max_background_flushes: -1 2023-07-10T16:54:18.695 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-07-10T16:54:18.695 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-07-10T16:54:18.695 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-10T16:54:18.696 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-10T16:54:18.696 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-07-10T16:54:18.696 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.advise_random_on_open: 1 2023-07-10T16:54:18.696 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.db_write_buffer_size: 0 2023-07-10T16:54:18.696 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.write_buffer_manager: 0x5599c5f43410 2023-07-10T16:54:18.696 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-10T16:54:18.696 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-07-10T16:54:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-10T16:54:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-07-10T16:54:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.rate_limiter: (nil) 2023-07-10T16:54:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-10T16:54:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.wal_recovery_mode: 2 2023-07-10T16:54:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.enable_thread_tracking: 0 2023-07-10T16:54:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.enable_pipelined_write: 0 2023-07-10T16:54:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.unordered_write: 0 2023-07-10T16:54:18.698 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-10T16:54:18.698 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-10T16:54:18.698 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-10T16:54:18.698 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-10T16:54:18.698 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.row_cache: None 2023-07-10T16:54:18.698 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.wal_filter: None 2023-07-10T16:54:18.698 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-10T16:54:18.699 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.allow_ingest_behind: 0 2023-07-10T16:54:18.699 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.preserve_deletes: 0 2023-07-10T16:54:18.699 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.two_write_queues: 0 2023-07-10T16:54:18.699 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.manual_wal_flush: 0 2023-07-10T16:54:18.699 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.atomic_flush: 0 2023-07-10T16:54:18.699 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-10T16:54:18.699 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-07-10T16:54:18.699 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-07-10T16:54:18.700 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.log_readahead_size: 0 2023-07-10T16:54:18.700 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-07-10T16:54:18.700 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.max_background_jobs: 2 2023-07-10T16:54:18.700 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.max_background_compactions: -1 2023-07-10T16:54:18.700 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-10T16:54:18.700 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-10T16:54:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-07-10T16:54:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.max_total_wal_size: 0 2023-07-10T16:54:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-10T16:54:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-07-10T16:54:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-07-10T16:54:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-10T16:54:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.max_open_files: -1 2023-07-10T16:54:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.bytes_per_sync: 0 2023-07-10T16:54:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-07-10T16:54:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-07-10T16:54:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.compaction_readahead_size: 0 2023-07-10T16:54:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Compression algorithms supported: 2023-07-10T16:54:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-10T16:54:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: kZSTD supported: 0 2023-07-10T16:54:18.703 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: kXpressCompression supported: 0 2023-07-10T16:54:18.703 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: kLZ4HCCompression supported: 1 2023-07-10T16:54:18.703 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: kLZ4Compression supported: 1 2023-07-10T16:54:18.703 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: kBZip2Compression supported: 0 2023-07-10T16:54:18.703 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: kZlibCompression supported: 1 2023-07-10T16:54:18.703 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: kSnappyCompression supported: 1 2023-07-10T16:54:18.703 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-07-10T16:54:18.703 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-07-10T16:54:18.704 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.704 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2023-07-10T16:54:18.704 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.704 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-07-10T16:54:18.704 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.704 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-10T16:54:18.704 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.merge_operator: 2023-07-10T16:54:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.compaction_filter: None 2023-07-10T16:54:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.compaction_filter_factory: None 2023-07-10T16:54:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-07-10T16:54:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.659+0000 7fbae8dca700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-07-10T16:54:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5599c5e500d0) 2023-07-10T16:54:18.706 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-07-10T16:54:18.706 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-10T16:54:18.706 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-10T16:54:18.706 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-07-10T16:54:18.706 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: index_type: 0 2023-07-10T16:54:18.706 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-07-10T16:54:18.706 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-07-10T16:54:18.707 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-07-10T16:54:18.707 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-07-10T16:54:18.707 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: checksum: 1 2023-07-10T16:54:18.707 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-07-10T16:54:18.707 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: block_cache: 0x5599c5e88d10 2023-07-10T16:54:18.707 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-07-10T16:54:18.707 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-07-10T16:54:18.707 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-07-10T16:54:18.708 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-07-10T16:54:18.708 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-07-10T16:54:18.708 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-07-10T16:54:18.708 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-07-10T16:54:18.708 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-07-10T16:54:18.708 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-07-10T16:54:18.708 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-07-10T16:54:18.709 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-07-10T16:54:18.709 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-07-10T16:54:18.709 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-07-10T16:54:18.709 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-07-10T16:54:18.709 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-07-10T16:54:18.710 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-07-10T16:54:18.710 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-07-10T16:54:18.710 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-07-10T16:54:18.710 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-07-10T16:54:18.710 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: format_version: 2 2023-07-10T16:54:18.710 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-07-10T16:54:18.710 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: block_align: 0 2023-07-10T16:54:18.710 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.711 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.write_buffer_size: 33554432 2023-07-10T16:54:18.711 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.max_write_buffer_number: 2 2023-07-10T16:54:18.711 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compression: NoCompression 2023-07-10T16:54:18.711 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.bottommost_compression: Disabled 2023-07-10T16:54:18.711 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.prefix_extractor: nullptr 2023-07-10T16:54:18.711 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-10T16:54:18.711 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.num_levels: 7 2023-07-10T16:54:18.711 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-10T16:54:18.712 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-10T16:54:18.712 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-10T16:54:18.712 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-10T16:54:18.712 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-10T16:54:18.712 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-10T16:54:18.712 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-10T16:54:18.713 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-10T16:54:18.713 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-10T16:54:18.713 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-07-10T16:54:18.713 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compression_opts.level: 32767 2023-07-10T16:54:18.713 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compression_opts.strategy: 0 2023-07-10T16:54:18.713 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-10T16:54:18.713 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-10T16:54:18.714 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compression_opts.enabled: false 2023-07-10T16:54:18.714 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-10T16:54:18.714 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-10T16:54:18.714 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-10T16:54:18.714 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.target_file_size_base: 67108864 2023-07-10T16:54:18.715 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-07-10T16:54:18.715 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-10T16:54:18.715 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-10T16:54:18.715 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-10T16:54:18.715 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-10T16:54:18.715 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-10T16:54:18.715 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-10T16:54:18.716 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-10T16:54:18.716 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-10T16:54:18.716 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-10T16:54:18.716 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-10T16:54:18.716 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-10T16:54:18.716 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-10T16:54:18.717 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.arena_block_size: 4194304 2023-07-10T16:54:18.717 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-10T16:54:18.717 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-10T16:54:18.717 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-07-10T16:54:18.717 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.disable_auto_compactions: 0 2023-07-10T16:54:18.717 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-10T16:54:18.717 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-10T16:54:18.718 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-10T16:54:18.718 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-10T16:54:18.718 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-10T16:54:18.718 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-10T16:54:18.718 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-10T16:54:18.718 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-10T16:54:18.718 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-10T16:54:18.718 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-10T16:54:18.719 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.table_properties_collectors: 2023-07-10T16:54:18.719 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.inplace_update_support: 0 2023-07-10T16:54:18.719 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-07-10T16:54:18.719 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-10T16:54:18.719 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-10T16:54:18.719 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-07-10T16:54:18.719 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.bloom_locality: 0 2023-07-10T16:54:18.719 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.max_successive_merges: 0 2023-07-10T16:54:18.720 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-07-10T16:54:18.720 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.paranoid_file_checks: 0 2023-07-10T16:54:18.720 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.force_consistency_checks: 0 2023-07-10T16:54:18.720 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.report_bg_io_stats: 0 2023-07-10T16:54:18.720 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.ttl: 2592000 2023-07-10T16:54:18.720 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-07-10T16:54:18.720 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-10T16:54:18.721 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.721 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-07-10T16:54:18.721 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.721 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbae8dca700 4 rocksdb: DB pointer 0x5599c5f4d800 2023-07-10T16:54:18.721 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbad1c65700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-07-10T16:54:18.721 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.663+0000 7fbad1c65700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-07-10T16:54:18.721 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-07-10T16:54:18.721 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-07-10T16:54:18.722 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-10T16:54:18.722 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-10T16:54:18.722 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-10T16:54:18.722 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-10T16:54:18.722 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-07-10T16:54:18.722 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-10T16:54:18.722 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-07-10T16:54:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-10T16:54:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-10T16:54:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-07-10T16:54:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-07-10T16:54:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-07-10T16:54:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-10T16:54:18.724 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-10T16:54:18.724 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-07-10T16:54:18.724 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-07-10T16:54:18.724 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-10T16:54:18.724 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-07-10T16:54:18.724 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-07-10T16:54:18.724 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-07-10T16:54:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-10T16:54:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-10T16:54:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-10T16:54:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-07-10T16:54:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-07-10T16:54:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-10T16:54:18.726 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-10T16:54:18.726 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-07-10T16:54:18.726 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-07-10T16:54:18.726 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.726 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-07-10T16:54:18.726 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-10T16:54:18.726 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-10T16:54:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-07-10T16:54:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-07-10T16:54:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-10T16:54:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-07-10T16:54:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-07-10T16:54:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-07-10T16:54:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-10T16:54:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-10T16:54:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-10T16:54:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-07-10T16:54:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: 2023-07-10T16:54:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.667+0000 7fbae8dca700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-07-10T16:54:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.671+0000 7fbae8dca700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-07-10T16:54:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph-mon: debug 2023-07-10T16:54:18.671+0000 7fbae8dca700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2023-07-10T16:54:19.295 INFO:teuthology.orchestra.run.smithi116.stderr:create mon.a on 2023-07-10T16:54:19.295 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:54:19.303 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl disable ceph.target 2023-07-10T16:54:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl enable ceph.target 2023-07-10T16:54:19.518 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2023-07-10T16:54:19.709 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl start ceph.target 2023-07-10T16:54:19.717 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl enable ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c.target 2023-07-10T16:54:19.725 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c.target -> /etc/systemd/system/ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c.target. 2023-07-10T16:54:19.725 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c.target -> /etc/systemd/system/ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c.target. 2023-07-10T16:54:19.910 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl start ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c.target 2023-07-10T16:54:19.919 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl daemon-reload 2023-07-10T16:54:20.106 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl stop ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.a 2023-07-10T16:54:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl reset-failed ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.a 2023-07-10T16:54:20.128 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: Failed to reset failed state of unit ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.a.service: Unit ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.a.service not loaded. 2023-07-10T16:54:20.128 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl enable ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.a 2023-07-10T16:54:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: Created symlink /etc/systemd/system/ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c.target.wants/ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.a.service -> /etc/systemd/system/ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@.service. 2023-07-10T16:54:20.313 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl start ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.a 2023-07-10T16:54:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:firewalld does not appear to be present 2023-07-10T16:54:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:Not possible to enable service . firewalld.service is not available 2023-07-10T16:54:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-07-10T16:54:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:Waiting for mon to start... 2023-07-10T16:54:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:Waiting for mon... 2023-07-10T16:54:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 status 2023-07-10T16:54:21.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:21 smithi116 bash[23284]: cluster 2023-07-10T16:54:21.701264+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2023-07-10T16:54:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: cluster: 2023-07-10T16:54:21.955 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: id: 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:54:21.955 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-07-10T16:54:21.955 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: 2023-07-10T16:54:21.955 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: services: 2023-07-10T16:54:21.956 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.239705s) 2023-07-10T16:54:21.956 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: mgr: no daemons active 2023-07-10T16:54:21.957 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-07-10T16:54:21.957 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: 2023-07-10T16:54:21.957 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: data: 2023-07-10T16:54:21.957 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-07-10T16:54:21.957 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-07-10T16:54:21.957 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-07-10T16:54:21.957 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: pgs: 2023-07-10T16:54:21.958 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: 2023-07-10T16:54:22.603 INFO:teuthology.orchestra.run.smithi116.stderr:mon is available 2023-07-10T16:54:22.603 INFO:teuthology.orchestra.run.smithi116.stderr:Assimilating anything we can from ceph.conf... 2023-07-10T16:54:22.603 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z -v /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z quay.io/ceph/ceph:v16.2.0 config assimilate-conf -i /var/lib/ceph/mon/ceph-a/config 2023-07-10T16:54:22.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:22 smithi116 bash[23284]: cluster 2023-07-10T16:54:21.707910+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2023-07-10T16:54:22.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:22 smithi116 bash[23284]: cluster 2023-07-10T16:54:21.708137+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0]} 2023-07-10T16:54:22.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:22 smithi116 bash[23284]: cluster 2023-07-10T16:54:21.711812+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2023-07-10T16:54:22.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:22 smithi116 bash[23284]: cluster 2023-07-10T16:54:21.716452+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-07-10T16:54:22.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:22 smithi116 bash[23284]: cluster 2023-07-10T16:54:21.717123+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-07-10T16:54:22.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:22 smithi116 bash[23284]: audit 2023-07-10T16:54:21.947550+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.116:0/2800121061' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-10T16:54:23.757 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: 2023-07-10T16:54:23.757 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: [global] 2023-07-10T16:54:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: fsid = 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:54:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: log_to_journald = false 2023-07-10T16:54:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.116:3300,v1:172.21.15.116:6789] 2023-07-10T16:54:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-07-10T16:54:23.760 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-07-10T16:54:23.760 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-07-10T16:54:23.760 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-07-10T16:54:23.760 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: 2023-07-10T16:54:23.760 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: [mgr] 2023-07-10T16:54:23.761 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2023-07-10T16:54:23.761 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-07-10T16:54:23.761 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: 2023-07-10T16:54:23.761 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: [mon] 2023-07-10T16:54:23.761 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: mon_warn_on_insecure_global_id_reclaim = false 2023-07-10T16:54:23.761 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-07-10T16:54:23.761 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: 2023-07-10T16:54:23.762 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: [osd] 2023-07-10T16:54:23.762 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-07-10T16:54:23.762 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-07-10T16:54:23.762 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-07-10T16:54:24.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:23 smithi116 bash[23284]: audit 2023-07-10T16:54:23.743429+0000 mon.a (mon.0) 8 : audit [INF] from='client.? 172.21.15.116:0/1177250774' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-10T16:54:24.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:23 smithi116 bash[23284]: audit 2023-07-10T16:54:23.750587+0000 mon.a (mon.0) 9 : audit [INF] from='client.? 172.21.15.116:0/1177250774' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-10T16:54:24.409 INFO:teuthology.orchestra.run.smithi116.stderr:Generating new minimal ceph.conf... 2023-07-10T16:54:24.410 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z -v /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z quay.io/ceph/ceph:v16.2.0 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-a/config 2023-07-10T16:54:25.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:25 smithi116 bash[23284]: audit 2023-07-10T16:54:25.542801+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.116:0/3713082330' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:54:26.157 INFO:teuthology.orchestra.run.smithi116.stderr:Restarting the monitor... 2023-07-10T16:54:26.157 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl restart ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.a 2023-07-10T16:54:26.506 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:26 smithi116 systemd[1]: Stopping Ceph mon.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T16:54:26.507 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:26 smithi116 bash[23284]: debug 2023-07-10T16:54:26.246+0000 7f12d88f7700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-07-10T16:54:26.507 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:26 smithi116 bash[23284]: debug 2023-07-10T16:54:26.246+0000 7f12d88f7700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-10T16:54:27.065 INFO:teuthology.orchestra.run.smithi116.stderr:Setting mon public_network to 172.21.0.0/20 2023-07-10T16:54:27.066 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config set mon public_network 172.21.0.0/20 2023-07-10T16:54:27.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:27 smithi116 docker[23675]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-mon.a 2023-07-10T16:54:27.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:27 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.a.service: Succeeded. 2023-07-10T16:54:27.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:27 smithi116 systemd[1]: Stopped Ceph mon.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T16:54:27.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:27 smithi116 systemd[1]: Started Ceph mon.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T16:54:28.607 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.306+0000 7f30d1838700 0 set uid:gid to 167:167 (ceph:ceph) 2023-07-10T16:54:28.608 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.306+0000 7f30d1838700 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-mon, pid 7 2023-07-10T16:54:28.608 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.306+0000 7f30d1838700 0 pidfile_write: ignore empty --pid-file 2023-07-10T16:54:28.608 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 0 load: jerasure load: lrc load: isa 2023-07-10T16:54:28.608 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: RocksDB version: 6.8.1 2023-07-10T16:54:28.608 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-07-10T16:54:28.608 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Compile date Mar 30 2021 2023-07-10T16:54:28.608 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: DB SUMMARY 2023-07-10T16:54:28.609 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: CURRENT file: CURRENT 2023-07-10T16:54:28.609 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: IDENTITY file: IDENTITY 2023-07-10T16:54:28.609 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-07-10T16:54:28.609 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2023-07-10T16:54:28.609 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 83116 ; 2023-07-10T16:54:28.609 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.error_if_exists: 0 2023-07-10T16:54:28.609 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.create_if_missing: 0 2023-07-10T16:54:28.610 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.paranoid_checks: 1 2023-07-10T16:54:28.610 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.env: 0x5621668781c0 2023-07-10T16:54:28.610 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.fs: Posix File System 2023-07-10T16:54:28.610 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.info_log: 0x5621686aefe0 2023-07-10T16:54:28.610 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_file_opening_threads: 16 2023-07-10T16:54:28.610 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.statistics: (nil) 2023-07-10T16:54:28.610 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.use_fsync: 0 2023-07-10T16:54:28.611 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_log_file_size: 0 2023-07-10T16:54:28.611 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-10T16:54:28.611 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-07-10T16:54:28.611 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.keep_log_file_num: 1000 2023-07-10T16:54:28.611 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.recycle_log_file_num: 0 2023-07-10T16:54:28.611 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.allow_fallocate: 1 2023-07-10T16:54:28.611 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.allow_mmap_reads: 0 2023-07-10T16:54:28.612 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.allow_mmap_writes: 0 2023-07-10T16:54:28.612 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.use_direct_reads: 0 2023-07-10T16:54:28.612 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-10T16:54:28.612 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.create_missing_column_families: 0 2023-07-10T16:54:28.612 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.db_log_dir: 2023-07-10T16:54:28.612 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2023-07-10T16:54:28.612 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-07-10T16:54:28.612 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_subcompactions: 1 2023-07-10T16:54:28.613 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_background_flushes: -1 2023-07-10T16:54:28.613 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-07-10T16:54:28.613 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-07-10T16:54:28.613 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-10T16:54:28.613 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-10T16:54:28.613 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-07-10T16:54:28.613 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.advise_random_on_open: 1 2023-07-10T16:54:28.613 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.db_write_buffer_size: 0 2023-07-10T16:54:28.614 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.write_buffer_manager: 0x562168745e30 2023-07-10T16:54:28.614 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-10T16:54:28.614 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-07-10T16:54:28.614 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-10T16:54:28.614 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-07-10T16:54:28.614 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.rate_limiter: (nil) 2023-07-10T16:54:28.614 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-10T16:54:28.615 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.wal_recovery_mode: 2 2023-07-10T16:54:28.615 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.enable_thread_tracking: 0 2023-07-10T16:54:28.616 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.enable_pipelined_write: 0 2023-07-10T16:54:28.616 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.unordered_write: 0 2023-07-10T16:54:28.616 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-10T16:54:28.617 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-10T16:54:28.617 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-10T16:54:28.617 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-10T16:54:28.617 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.row_cache: None 2023-07-10T16:54:28.617 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.wal_filter: None 2023-07-10T16:54:28.617 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-10T16:54:28.617 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.allow_ingest_behind: 0 2023-07-10T16:54:28.618 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.preserve_deletes: 0 2023-07-10T16:54:28.618 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.two_write_queues: 0 2023-07-10T16:54:28.618 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.manual_wal_flush: 0 2023-07-10T16:54:28.618 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.atomic_flush: 0 2023-07-10T16:54:28.618 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-10T16:54:28.618 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-07-10T16:54:28.618 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-07-10T16:54:28.619 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.log_readahead_size: 0 2023-07-10T16:54:28.619 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-07-10T16:54:28.619 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_background_jobs: 2 2023-07-10T16:54:28.619 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_background_compactions: -1 2023-07-10T16:54:28.619 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-10T16:54:28.619 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-10T16:54:28.619 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-07-10T16:54:28.620 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_total_wal_size: 0 2023-07-10T16:54:28.620 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-10T16:54:28.620 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-07-10T16:54:28.620 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-07-10T16:54:28.620 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-10T16:54:28.620 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_open_files: -1 2023-07-10T16:54:28.620 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.bytes_per_sync: 0 2023-07-10T16:54:28.620 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-07-10T16:54:28.621 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-07-10T16:54:28.621 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compaction_readahead_size: 0 2023-07-10T16:54:28.621 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Compression algorithms supported: 2023-07-10T16:54:28.621 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-10T16:54:28.621 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: kZSTD supported: 0 2023-07-10T16:54:28.621 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: kXpressCompression supported: 0 2023-07-10T16:54:28.621 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: kLZ4HCCompression supported: 1 2023-07-10T16:54:28.622 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: kLZ4Compression supported: 1 2023-07-10T16:54:28.622 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: kBZip2Compression supported: 0 2023-07-10T16:54:28.622 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: kZlibCompression supported: 1 2023-07-10T16:54:28.622 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: kSnappyCompression supported: 1 2023-07-10T16:54:28.622 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-07-10T16:54:28.622 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2023-07-10T16:54:28.622 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-07-10T16:54:28.622 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-10T16:54:28.623 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.merge_operator: 2023-07-10T16:54:28.623 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compaction_filter: None 2023-07-10T16:54:28.623 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compaction_filter_factory: None 2023-07-10T16:54:28.623 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-07-10T16:54:28.623 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-07-10T16:54:28.623 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5621686520d8) 2023-07-10T16:54:28.623 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: cache_index_and_filter_blocks: 1 2023-07-10T16:54:28.623 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-10T16:54:28.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-10T16:54:28.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: pin_top_level_index_and_filter: 1 2023-07-10T16:54:28.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: index_type: 0 2023-07-10T16:54:28.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: data_block_index_type: 0 2023-07-10T16:54:28.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: index_shortening: 1 2023-07-10T16:54:28.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: data_block_hash_table_util_ratio: 0.750000 2023-07-10T16:54:28.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: hash_index_allow_collision: 1 2023-07-10T16:54:28.625 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: checksum: 1 2023-07-10T16:54:28.625 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: no_block_cache: 0 2023-07-10T16:54:28.625 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: block_cache: 0x56216868af10 2023-07-10T16:54:28.625 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: block_cache_name: BinnedLRUCache 2023-07-10T16:54:28.625 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: block_cache_options: 2023-07-10T16:54:28.625 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: capacity : 536870912 2023-07-10T16:54:28.625 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: num_shard_bits : 4 2023-07-10T16:54:28.625 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: strict_capacity_limit : 0 2023-07-10T16:54:28.626 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: high_pri_pool_ratio: 0.000 2023-07-10T16:54:28.626 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: block_cache_compressed: (nil) 2023-07-10T16:54:28.626 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: persistent_cache: (nil) 2023-07-10T16:54:28.626 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: block_size: 4096 2023-07-10T16:54:28.626 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: block_size_deviation: 10 2023-07-10T16:54:28.626 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: block_restart_interval: 16 2023-07-10T16:54:28.626 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: index_block_restart_interval: 1 2023-07-10T16:54:28.626 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: metadata_block_size: 4096 2023-07-10T16:54:28.627 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: partition_filters: 0 2023-07-10T16:54:28.627 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: use_delta_encoding: 1 2023-07-10T16:54:28.627 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: filter_policy: rocksdb.BuiltinBloomFilter 2023-07-10T16:54:28.627 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: whole_key_filtering: 1 2023-07-10T16:54:28.627 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: verify_compression: 0 2023-07-10T16:54:28.627 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: read_amp_bytes_per_bit: 0 2023-07-10T16:54:28.627 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: format_version: 2 2023-07-10T16:54:28.628 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: enable_index_compression: 1 2023-07-10T16:54:28.628 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: block_align: 0 2023-07-10T16:54:28.628 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.write_buffer_size: 33554432 2023-07-10T16:54:28.628 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_write_buffer_number: 2 2023-07-10T16:54:28.628 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compression: NoCompression 2023-07-10T16:54:28.628 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.bottommost_compression: Disabled 2023-07-10T16:54:28.628 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.prefix_extractor: nullptr 2023-07-10T16:54:28.629 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-10T16:54:28.629 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.num_levels: 7 2023-07-10T16:54:28.629 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-10T16:54:28.629 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-10T16:54:28.629 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-10T16:54:28.629 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-10T16:54:28.629 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-10T16:54:28.630 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-10T16:54:28.630 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-10T16:54:28.630 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-10T16:54:28.630 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-10T16:54:28.630 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-07-10T16:54:28.630 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compression_opts.level: 32767 2023-07-10T16:54:28.630 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compression_opts.strategy: 0 2023-07-10T16:54:28.631 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-10T16:54:28.631 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-10T16:54:28.631 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compression_opts.enabled: false 2023-07-10T16:54:28.631 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-10T16:54:28.631 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-10T16:54:28.631 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-10T16:54:28.632 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.target_file_size_base: 67108864 2023-07-10T16:54:28.632 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-07-10T16:54:28.632 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-10T16:54:28.632 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-10T16:54:28.632 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-10T16:54:28.632 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-10T16:54:28.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-10T16:54:28.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-10T16:54:28.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-10T16:54:28.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-10T16:54:28.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-10T16:54:28.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-10T16:54:28.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-10T16:54:28.634 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-10T16:54:28.634 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.arena_block_size: 4194304 2023-07-10T16:54:28.634 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-10T16:54:28.634 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-10T16:54:28.634 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-07-10T16:54:28.634 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.disable_auto_compactions: 0 2023-07-10T16:54:28.634 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-10T16:54:28.634 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-10T16:54:28.635 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-10T16:54:28.635 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-10T16:54:28.635 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-10T16:54:28.635 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-10T16:54:28.635 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-10T16:54:28.635 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-10T16:54:28.635 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-10T16:54:28.635 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-10T16:54:28.636 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.table_properties_collectors: 2023-07-10T16:54:28.636 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.inplace_update_support: 0 2023-07-10T16:54:28.636 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-07-10T16:54:28.636 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-10T16:54:28.636 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-10T16:54:28.637 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-07-10T16:54:28.637 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.bloom_locality: 0 2023-07-10T16:54:28.637 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.max_successive_merges: 0 2023-07-10T16:54:28.637 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-07-10T16:54:28.637 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.paranoid_file_checks: 0 2023-07-10T16:54:28.637 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.force_consistency_checks: 0 2023-07-10T16:54:28.637 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.report_bg_io_stats: 0 2023-07-10T16:54:28.638 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.ttl: 2592000 2023-07-10T16:54:28.638 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-07-10T16:54:28.638 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-10T16:54:28.638 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-07-10T16:54:28.638 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1689008068315122, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-07-10T16:54:28.638 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.310+0000 7f30d1838700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-07-10T16:54:28.638 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.314+0000 7f30d1838700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-07-10T16:54:28.639 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.314+0000 7f30d1838700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1689008068316285, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 80084, "table_properties": {"data_size": 78342, "index_size": 347, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 8979, "raw_average_key_size": 46, "raw_value_size": 73101, "raw_average_value_size": 376, "num_data_blocks": 11, "num_entries": 194, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1689008068, "oldest_key_time": 3, "file_creation_time": 0}} 2023-07-10T16:54:28.639 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.314+0000 7f30d1838700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-07-10T16:54:28.639 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.314+0000 7f30d1838700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1689008068317162, "job": 1, "event": "recovery_finished"} 2023-07-10T16:54:28.639 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.314+0000 7f30d1838700 4 rocksdb: DB pointer 0x56216874f800 2023-07-10T16:54:28.639 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.314+0000 7f30b8652700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-07-10T16:54:28.639 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.314+0000 7f30b8652700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-07-10T16:54:28.639 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: ** DB Stats ** 2023-07-10T16:54:28.639 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Uptime(secs): 0.0 total, 0.0 interval 2023-07-10T16:54:28.640 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-10T16:54:28.640 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-10T16:54:28.640 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-10T16:54:28.640 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-10T16:54:28.640 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-07-10T16:54:28.640 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-10T16:54:28.640 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: ** Compaction Stats [default] ** 2023-07-10T16:54:28.641 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-10T16:54:28.641 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-10T16:54:28.641 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: L0 2/0 79.86 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 100.1 0.00 0.00 1 0.001 0 0 2023-07-10T16:54:28.641 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Sum 2/0 79.86 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 100.1 0.00 0.00 1 0.001 0 0 2023-07-10T16:54:28.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 100.1 0.00 0.00 1 0.001 0 0 2023-07-10T16:54:28.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: ** Compaction Stats [default] ** 2023-07-10T16:54:28.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-10T16:54:28.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-10T16:54:28.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 100.1 0.00 0.00 1 0.001 0 0 2023-07-10T16:54:28.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Uptime(secs): 0.0 total, 0.0 interval 2023-07-10T16:54:28.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Flush(GB): cumulative 0.000, interval 0.000 2023-07-10T16:54:28.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-10T16:54:28.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: AddFile(Total Files): cumulative 0, interval 0 2023-07-10T16:54:28.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: AddFile(L0 Files): cumulative 0, interval 0 2023-07-10T16:54:28.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: AddFile(Keys): cumulative 0, interval 0 2023-07-10T16:54:28.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Cumulative compaction: 0.00 GB write, 17.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-10T16:54:28.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Interval compaction: 0.00 GB write, 17.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-10T16:54:28.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-10T16:54:28.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: ** File Read Latency Histogram By Level [default] ** 2023-07-10T16:54:28.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: ** Compaction Stats [default] ** 2023-07-10T16:54:28.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-10T16:54:28.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-10T16:54:28.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: L0 2/0 79.86 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 100.1 0.00 0.00 1 0.001 0 0 2023-07-10T16:54:28.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Sum 2/0 79.86 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 100.1 0.00 0.00 1 0.001 0 0 2023-07-10T16:54:28.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-07-10T16:54:28.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: ** Compaction Stats [default] ** 2023-07-10T16:54:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-07-10T16:54:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-10T16:54:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 100.1 0.00 0.00 1 0.001 0 0 2023-07-10T16:54:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Uptime(secs): 0.0 total, 0.0 interval 2023-07-10T16:54:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Flush(GB): cumulative 0.000, interval 0.000 2023-07-10T16:54:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-10T16:54:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: AddFile(Total Files): cumulative 0, interval 0 2023-07-10T16:54:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: AddFile(L0 Files): cumulative 0, interval 0 2023-07-10T16:54:28.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: AddFile(Keys): cumulative 0, interval 0 2023-07-10T16:54:28.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Cumulative compaction: 0.00 GB write, 17.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-10T16:54:28.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-10T16:54:28.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-10T16:54:28.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: ** File Read Latency Histogram By Level [default] ** 2023-07-10T16:54:28.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.318+0000 7f30d1838700 0 starting mon.a rank 0 at public addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] at bind addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:54:28.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.318+0000 7f30d1838700 1 mon.a@-1(???) e1 preinit fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:54:28.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.318+0000 7f30d1838700 0 mon.a@-1(???).mds e1 new map 2023-07-10T16:54:28.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.318+0000 7f30d1838700 0 mon.a@-1(???).mds e1 print_map 2023-07-10T16:54:28.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: e1 2023-07-10T16:54:28.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: enable_multiple, ever_enabled_multiple: 1,1 2023-07-10T16:54:28.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-10T16:54:28.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: legacy client fscid: -1 2023-07-10T16:54:28.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: 2023-07-10T16:54:28.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: No filesystems configured 2023-07-10T16:54:28.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.318+0000 7f30d1838700 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-10T16:54:28.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.318+0000 7f30d1838700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-10T16:54:28.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.318+0000 7f30d1838700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-10T16:54:28.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.318+0000 7f30d1838700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-10T16:54:28.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: audit 2023-07-10T16:54:25.542801+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.116:0/3713082330' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:54:28.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.318+0000 7f30d1838700 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-10T16:54:28.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.322+0000 7f30d1838700 20 expand_channel_meta expand map: {default=false} 2023-07-10T16:54:28.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.322+0000 7f30d1838700 20 expand_channel_meta from 'false' to 'false' 2023-07-10T16:54:28.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.322+0000 7f30d1838700 20 expand_channel_meta expanded map: {default=false} 2023-07-10T16:54:28.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.322+0000 7f30d1838700 20 expand_channel_meta expand map: {default=info} 2023-07-10T16:54:28.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.322+0000 7f30d1838700 20 expand_channel_meta from 'info' to 'info' 2023-07-10T16:54:28.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.322+0000 7f30d1838700 20 expand_channel_meta expanded map: {default=info} 2023-07-10T16:54:28.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.322+0000 7f30d1838700 20 expand_channel_meta expand map: {default=daemon} 2023-07-10T16:54:28.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.322+0000 7f30d1838700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-07-10T16:54:28.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.322+0000 7f30d1838700 20 expand_channel_meta expanded map: {default=daemon} 2023-07-10T16:54:28.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.322+0000 7f30d1838700 20 expand_channel_meta expand map: {default=debug} 2023-07-10T16:54:28.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.322+0000 7f30d1838700 20 expand_channel_meta from 'debug' to 'debug' 2023-07-10T16:54:28.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: debug 2023-07-10T16:54:28.322+0000 7f30d1838700 20 expand_channel_meta expanded map: {default=debug} 2023-07-10T16:54:28.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: cluster 2023-07-10T16:54:28.328766+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2023-07-10T16:54:28.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: cluster 2023-07-10T16:54:28.328858+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0]} 2023-07-10T16:54:28.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: cluster 2023-07-10T16:54:28.328924+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2023-07-10T16:54:28.651 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: cluster 2023-07-10T16:54:28.328966+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-07-10T16:54:28.651 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:28 smithi116 bash[23740]: cluster 2023-07-10T16:54:28.329462+0000 mon.a (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-07-10T16:54:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:Wrote config to /etc/ceph/ceph.conf 2023-07-10T16:54:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-10T16:54:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:Creating mgr... 2023-07-10T16:54:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:Verifying port 9283 ... 2023-07-10T16:54:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl daemon-reload 2023-07-10T16:54:29.447 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl stop ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.y 2023-07-10T16:54:29.459 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl reset-failed ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.y 2023-07-10T16:54:29.463 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: Failed to reset failed state of unit ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.y.service: Unit ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.y.service not loaded. 2023-07-10T16:54:29.464 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl enable ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.y 2023-07-10T16:54:29.470 INFO:teuthology.orchestra.run.smithi116.stderr:systemctl: Created symlink /etc/systemd/system/ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c.target.wants/ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.y.service -> /etc/systemd/system/ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@.service. 2023-07-10T16:54:29.664 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: systemctl start ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.y 2023-07-10T16:54:29.678 INFO:teuthology.orchestra.run.smithi116.stderr:firewalld does not appear to be present 2023-07-10T16:54:29.678 INFO:teuthology.orchestra.run.smithi116.stderr:Not possible to enable service . firewalld.service is not available 2023-07-10T16:54:29.678 INFO:teuthology.orchestra.run.smithi116.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-07-10T16:54:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:firewalld does not appear to be present 2023-07-10T16:54:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-07-10T16:54:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:Waiting for mgr to start... 2023-07-10T16:54:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:Waiting for mgr... 2023-07-10T16:54:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 status -f json-pretty 2023-07-10T16:54:29.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:29 smithi116 bash[23740]: audit 2023-07-10T16:54:28.576695+0000 mon.a (mon.0) 6 : audit [INF] from='client.? 172.21.15.116:0/738506195' entity='client.admin' 2023-07-10T16:54:31.290 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: 2023-07-10T16:54:31.291 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: { 2023-07-10T16:54:31.291 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "fsid": "3f83c2ec-1f42-11ee-9b31-001a4aab830c", 2023-07-10T16:54:31.291 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "health": { 2023-07-10T16:54:31.291 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-07-10T16:54:31.292 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "checks": {}, 2023-07-10T16:54:31.292 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "mutes": [] 2023-07-10T16:54:31.293 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:31.293 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-07-10T16:54:31.293 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "quorum": [ 2023-07-10T16:54:31.294 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: 0 2023-07-10T16:54:31.295 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: ], 2023-07-10T16:54:31.295 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "quorum_names": [ 2023-07-10T16:54:31.295 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "a" 2023-07-10T16:54:31.295 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: ], 2023-07-10T16:54:31.295 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "quorum_age": 2, 2023-07-10T16:54:31.295 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "monmap": { 2023-07-10T16:54:31.295 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-10T16:54:31.295 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-07-10T16:54:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_mons": 1 2023-07-10T16:54:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "osdmap": { 2023-07-10T16:54:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-10T16:54:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_osds": 0, 2023-07-10T16:54:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-07-10T16:54:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-07-10T16:54:31.297 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-07-10T16:54:31.297 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-07-10T16:54:31.297 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-07-10T16:54:31.299 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:31.299 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "pgmap": { 2023-07-10T16:54:31.300 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-07-10T16:54:31.300 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-07-10T16:54:31.300 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_pools": 0, 2023-07-10T16:54:31.300 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_objects": 0, 2023-07-10T16:54:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-07-10T16:54:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-07-10T16:54:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-07-10T16:54:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "bytes_total": 0 2023-07-10T16:54:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:31.302 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "fsmap": { 2023-07-10T16:54:31.302 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-10T16:54:31.302 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "by_rank": [], 2023-07-10T16:54:31.302 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "up:standby": 0 2023-07-10T16:54:31.302 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:31.302 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "mgrmap": { 2023-07-10T16:54:31.302 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "available": false, 2023-07-10T16:54:31.303 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-07-10T16:54:31.303 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "modules": [ 2023-07-10T16:54:31.303 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "iostat", 2023-07-10T16:54:31.303 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "restful" 2023-07-10T16:54:31.303 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: ], 2023-07-10T16:54:31.303 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "services": {} 2023-07-10T16:54:31.304 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:31.304 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "servicemap": { 2023-07-10T16:54:31.304 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-10T16:54:31.304 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "modified": "2023-07-10T16:54:21.709801+0000", 2023-07-10T16:54:31.304 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "services": {} 2023-07-10T16:54:31.304 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:31.304 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "progress_events": {} 2023-07-10T16:54:31.304 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: } 2023-07-10T16:54:31.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:31 smithi116 bash[23740]: audit 2023-07-10T16:54:31.283922+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.116:0/1013033573' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-10T16:54:31.942 INFO:teuthology.orchestra.run.smithi116.stderr:mgr not available, waiting (1/15)... 2023-07-10T16:54:33.944 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 status -f json-pretty 2023-07-10T16:54:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: 2023-07-10T16:54:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: { 2023-07-10T16:54:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "fsid": "3f83c2ec-1f42-11ee-9b31-001a4aab830c", 2023-07-10T16:54:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "health": { 2023-07-10T16:54:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-07-10T16:54:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "checks": {}, 2023-07-10T16:54:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "mutes": [] 2023-07-10T16:54:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-07-10T16:54:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "quorum": [ 2023-07-10T16:54:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: 0 2023-07-10T16:54:35.093 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: ], 2023-07-10T16:54:35.093 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "quorum_names": [ 2023-07-10T16:54:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "a" 2023-07-10T16:54:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: ], 2023-07-10T16:54:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "quorum_age": 6, 2023-07-10T16:54:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "monmap": { 2023-07-10T16:54:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-10T16:54:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-07-10T16:54:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_mons": 1 2023-07-10T16:54:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "osdmap": { 2023-07-10T16:54:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-10T16:54:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_osds": 0, 2023-07-10T16:54:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-07-10T16:54:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-07-10T16:54:35.100 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-07-10T16:54:35.100 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-07-10T16:54:35.100 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-07-10T16:54:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "pgmap": { 2023-07-10T16:54:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-07-10T16:54:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-07-10T16:54:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_pools": 0, 2023-07-10T16:54:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_objects": 0, 2023-07-10T16:54:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-07-10T16:54:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-07-10T16:54:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-07-10T16:54:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "bytes_total": 0 2023-07-10T16:54:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "fsmap": { 2023-07-10T16:54:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-10T16:54:35.103 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "by_rank": [], 2023-07-10T16:54:35.103 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "up:standby": 0 2023-07-10T16:54:35.104 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:35.104 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "mgrmap": { 2023-07-10T16:54:35.104 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "available": false, 2023-07-10T16:54:35.104 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-07-10T16:54:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "modules": [ 2023-07-10T16:54:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "iostat", 2023-07-10T16:54:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "restful" 2023-07-10T16:54:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: ], 2023-07-10T16:54:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "services": {} 2023-07-10T16:54:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "servicemap": { 2023-07-10T16:54:35.106 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-10T16:54:35.106 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "modified": "2023-07-10T16:54:21.709801+0000", 2023-07-10T16:54:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "services": {} 2023-07-10T16:54:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "progress_events": {} 2023-07-10T16:54:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: } 2023-07-10T16:54:35.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:35 smithi116 bash[23740]: audit 2023-07-10T16:54:35.082122+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? 172.21.15.116:0/1797262905' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-10T16:54:35.713 INFO:teuthology.orchestra.run.smithi116.stderr:mgr not available, waiting (2/15)... 2023-07-10T16:54:37.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:36 smithi116 bash[23740]: cluster 2023-07-10T16:54:36.857446+0000 mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2023-07-10T16:54:37.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:36 smithi116 bash[23740]: cluster 2023-07-10T16:54:36.861571+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.0043606s) 2023-07-10T16:54:37.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:36 smithi116 bash[23740]: audit 2023-07-10T16:54:36.865457+0000 mon.a (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.116:0/212646445' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-10T16:54:37.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:36 smithi116 bash[23740]: audit 2023-07-10T16:54:36.866145+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.116:0/212646445' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-10T16:54:37.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:36 smithi116 bash[23740]: audit 2023-07-10T16:54:36.866788+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.116:0/212646445' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-10T16:54:37.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:36 smithi116 bash[23740]: audit 2023-07-10T16:54:36.867207+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.116:0/212646445' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T16:54:37.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:36 smithi116 bash[23740]: audit 2023-07-10T16:54:36.867647+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.116:0/212646445' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-10T16:54:37.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:36 smithi116 bash[23740]: cluster 2023-07-10T16:54:36.872347+0000 mon.a (mon.0) 16 : cluster [INF] Manager daemon y is now available 2023-07-10T16:54:37.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:36 smithi116 bash[23740]: audit 2023-07-10T16:54:36.894385+0000 mon.a (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.116:0/212646445' entity='mgr.y' 2023-07-10T16:54:37.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:36 smithi116 bash[23740]: audit 2023-07-10T16:54:36.897650+0000 mon.a (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.116:0/212646445' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T16:54:37.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:36 smithi116 bash[23740]: audit 2023-07-10T16:54:36.900089+0000 mon.a (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.116:0/212646445' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T16:54:37.716 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 status -f json-pretty 2023-07-10T16:54:38.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:37 smithi116 bash[23740]: audit 2023-07-10T16:54:36.912109+0000 mon.a (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.116:0/212646445' entity='mgr.y' 2023-07-10T16:54:38.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:37 smithi116 bash[23740]: audit 2023-07-10T16:54:36.921266+0000 mon.a (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.116:0/212646445' entity='mgr.y' 2023-07-10T16:54:38.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:37 smithi116 bash[23740]: cluster 2023-07-10T16:54:37.868875+0000 mon.a (mon.0) 22 : cluster [DBG] mgrmap e3: y(active, since 1.01164s) 2023-07-10T16:54:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: 2023-07-10T16:54:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: { 2023-07-10T16:54:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "fsid": "3f83c2ec-1f42-11ee-9b31-001a4aab830c", 2023-07-10T16:54:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "health": { 2023-07-10T16:54:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-07-10T16:54:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "checks": {}, 2023-07-10T16:54:39.199 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "mutes": [] 2023-07-10T16:54:39.199 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:39.199 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-07-10T16:54:39.199 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "quorum": [ 2023-07-10T16:54:39.199 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: 0 2023-07-10T16:54:39.199 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: ], 2023-07-10T16:54:39.199 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "quorum_names": [ 2023-07-10T16:54:39.199 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "a" 2023-07-10T16:54:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: ], 2023-07-10T16:54:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "quorum_age": 10, 2023-07-10T16:54:39.201 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "monmap": { 2023-07-10T16:54:39.201 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-10T16:54:39.201 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-07-10T16:54:39.201 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_mons": 1 2023-07-10T16:54:39.201 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "osdmap": { 2023-07-10T16:54:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-10T16:54:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_osds": 0, 2023-07-10T16:54:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-07-10T16:54:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-07-10T16:54:39.205 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-07-10T16:54:39.205 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-07-10T16:54:39.205 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-07-10T16:54:39.205 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:39.205 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "pgmap": { 2023-07-10T16:54:39.205 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-07-10T16:54:39.206 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-07-10T16:54:39.206 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_pools": 0, 2023-07-10T16:54:39.206 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_objects": 0, 2023-07-10T16:54:39.207 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-07-10T16:54:39.207 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-07-10T16:54:39.207 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-07-10T16:54:39.207 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "bytes_total": 0 2023-07-10T16:54:39.207 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:39.208 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "fsmap": { 2023-07-10T16:54:39.208 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-10T16:54:39.209 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "by_rank": [], 2023-07-10T16:54:39.209 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "up:standby": 0 2023-07-10T16:54:39.209 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:39.209 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "mgrmap": { 2023-07-10T16:54:39.209 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "available": true, 2023-07-10T16:54:39.209 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-07-10T16:54:39.209 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "modules": [ 2023-07-10T16:54:39.210 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "iostat", 2023-07-10T16:54:39.210 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "restful" 2023-07-10T16:54:39.210 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: ], 2023-07-10T16:54:39.210 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "services": {} 2023-07-10T16:54:39.210 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:39.210 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "servicemap": { 2023-07-10T16:54:39.211 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "epoch": 1, 2023-07-10T16:54:39.212 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "modified": "2023-07-10T16:54:21.709801+0000", 2023-07-10T16:54:39.212 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "services": {} 2023-07-10T16:54:39.212 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: }, 2023-07-10T16:54:39.212 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "progress_events": {} 2023-07-10T16:54:39.212 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: } 2023-07-10T16:54:39.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:39 smithi116 bash[23740]: audit 2023-07-10T16:54:39.187813+0000 mon.a (mon.0) 23 : audit [DBG] from='client.? 172.21.15.116:0/3663537097' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-10T16:54:39.807 INFO:teuthology.orchestra.run.smithi116.stderr:mgr is available 2023-07-10T16:54:39.812 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config-key set mgr/cephadm/ssh_user root 2023-07-10T16:54:40.599 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:40 smithi116 bash[23740]: cluster 2023-07-10T16:54:39.343735+0000 mon.a (mon.0) 24 : cluster [DBG] mgrmap e4: y(active, since 2s) 2023-07-10T16:54:41.277 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: set mgr/cephadm/ssh_user 2023-07-10T16:54:41.794 INFO:teuthology.orchestra.run.smithi116.stderr:Enabling cephadm module... 2023-07-10T16:54:41.795 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 mgr module enable cephadm 2023-07-10T16:54:42.561 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:42 smithi116 bash[23740]: audit 2023-07-10T16:54:41.266934+0000 mon.a (mon.0) 25 : audit [INF] from='client.? 172.21.15.116:0/2715744487' entity='client.admin' 2023-07-10T16:54:43.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:43 smithi116 bash[23740]: audit 2023-07-10T16:54:43.176449+0000 mon.a (mon.0) 26 : audit [INF] from='client.? 172.21.15.116:0/693002548' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-10T16:54:43.646 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:43 smithi116 bash[24013]: ignoring --setgroup ceph since I am not root 2023-07-10T16:54:43.908 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 mgr stat 2023-07-10T16:54:44.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:44 smithi116 bash[23740]: audit 2023-07-10T16:54:43.275047+0000 mon.a (mon.0) 27 : audit [INF] from='client.? 172.21.15.116:0/693002548' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-10T16:54:44.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:44 smithi116 bash[23740]: cluster 2023-07-10T16:54:43.275184+0000 mon.a (mon.0) 28 : cluster [DBG] mgrmap e5: y(active, since 6s) 2023-07-10T16:54:45.276 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: { 2023-07-10T16:54:45.276 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "epoch": 5, 2023-07-10T16:54:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "available": true, 2023-07-10T16:54:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "active_name": "y", 2023-07-10T16:54:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_standby": 0 2023-07-10T16:54:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: } 2023-07-10T16:54:45.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:45 smithi116 bash[23740]: audit 2023-07-10T16:54:45.268611+0000 mon.a (mon.0) 29 : audit [DBG] from='client.? 172.21.15.116:0/4137673202' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-10T16:54:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:Waiting for the mgr to restart... 2023-07-10T16:54:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:Waiting for mgr epoch 5... 2023-07-10T16:54:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 tell mgr mgr_status 2023-07-10T16:54:50.362 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: { 2023-07-10T16:54:50.363 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-07-10T16:54:50.363 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "initialized": true 2023-07-10T16:54:50.363 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: } 2023-07-10T16:54:50.413 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: cluster 2023-07-10T16:54:49.149239+0000 mon.a (mon.0) 30 : cluster [INF] Active manager daemon y restarted 2023-07-10T16:54:50.413 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: cluster 2023-07-10T16:54:49.150098+0000 mon.a (mon.0) 31 : cluster [INF] Activating manager daemon y 2023-07-10T16:54:50.413 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: cluster 2023-07-10T16:54:49.153686+0000 mon.a (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-07-10T16:54:50.414 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: cluster 2023-07-10T16:54:49.336746+0000 mon.a (mon.0) 33 : cluster [DBG] mgrmap e6: y(active, starting, since 0.186868s) 2023-07-10T16:54:50.414 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: audit 2023-07-10T16:54:49.338776+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T16:54:50.414 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: audit 2023-07-10T16:54:49.340961+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-10T16:54:50.414 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: audit 2023-07-10T16:54:49.342325+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-10T16:54:50.414 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: audit 2023-07-10T16:54:49.342898+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-10T16:54:50.415 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: audit 2023-07-10T16:54:49.343145+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-10T16:54:50.415 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: cluster 2023-07-10T16:54:49.347323+0000 mon.a (mon.0) 39 : cluster [INF] Manager daemon y is now available 2023-07-10T16:54:50.415 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: audit 2023-07-10T16:54:49.367872+0000 mon.a (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' 2023-07-10T16:54:50.415 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: audit 2023-07-10T16:54:49.374122+0000 mon.a (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' 2023-07-10T16:54:50.415 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: audit 2023-07-10T16:54:49.378304+0000 mon.a (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' 2023-07-10T16:54:50.415 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: audit 2023-07-10T16:54:49.517222+0000 mon.a (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' 2023-07-10T16:54:50.415 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: audit 2023-07-10T16:54:49.518583+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:54:50.416 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: audit 2023-07-10T16:54:49.521451+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:54:50.416 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: audit 2023-07-10T16:54:49.525707+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:54:50.416 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: audit 2023-07-10T16:54:49.549792+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T16:54:50.416 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:50 smithi116 bash[23740]: audit 2023-07-10T16:54:49.552556+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T16:54:51.029 INFO:teuthology.orchestra.run.smithi116.stderr:mgr epoch 5 is available 2023-07-10T16:54:51.029 INFO:teuthology.orchestra.run.smithi116.stderr:Setting orchestrator backend to cephadm... 2023-07-10T16:54:51.029 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 orch set backend cephadm 2023-07-10T16:54:51.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:51 smithi116 bash[23740]: audit 2023-07-10T16:54:50.339789+0000 mgr.y (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-10T16:54:51.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:51 smithi116 bash[23740]: cluster 2023-07-10T16:54:50.340767+0000 mon.a (mon.0) 49 : cluster [DBG] mgrmap e7: y(active, since 1.19088s) 2023-07-10T16:54:51.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:51 smithi116 bash[23740]: audit 2023-07-10T16:54:50.355118+0000 mgr.y (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-10T16:54:52.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:52 smithi116 bash[23740]: cluster 2023-07-10T16:54:51.341383+0000 mon.a (mon.0) 50 : cluster [DBG] mgrmap e8: y(active, since 2s) 2023-07-10T16:54:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:Generating ssh key... 2023-07-10T16:54:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 cephadm generate-key 2023-07-10T16:54:53.800 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:53 smithi116 bash[23740]: audit 2023-07-10T16:54:52.394637+0000 mgr.y (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:54:53.800 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:53 smithi116 bash[23740]: audit 2023-07-10T16:54:52.402094+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' 2023-07-10T16:54:53.800 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:53 smithi116 bash[23740]: audit 2023-07-10T16:54:52.409473+0000 mon.a (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:54:54.896 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: Generating public/private rsa key pair. 2023-07-10T16:54:54.896 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: Your identification has been saved in /tmp/tmpi01qbre8/key. 2023-07-10T16:54:54.896 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: Your public key has been saved in /tmp/tmpi01qbre8/key.pub. 2023-07-10T16:54:54.897 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: The key fingerprint is: 2023-07-10T16:54:54.897 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: SHA256:1/h1cYPWRxTG69y4fdIktaytN5SQVNzR6RRg/JRhG08 ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:54:54.897 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: The key's randomart image is: 2023-07-10T16:54:54.898 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: +---[RSA 3072]----+ 2023-07-10T16:54:54.898 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: | .o=@E| 2023-07-10T16:54:54.898 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: | .o=X*| 2023-07-10T16:54:54.898 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: | .o*==| 2023-07-10T16:54:54.899 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: | o.o +*| 2023-07-10T16:54:54.899 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: | S o . *o=| 2023-07-10T16:54:54.899 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: | . . .oOo| 2023-07-10T16:54:54.900 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: | . +* | 2023-07-10T16:54:54.900 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: | .o++| 2023-07-10T16:54:54.900 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: | .o.o| 2023-07-10T16:54:54.901 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:54 smithi116 bash[24013]: +----[SHA256]-----+ 2023-07-10T16:54:55.186 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 cephadm get-pub-key 2023-07-10T16:54:55.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:55 smithi116 bash[23740]: cephadm 2023-07-10T16:54:54.328515+0000 mgr.y (mgr.14116) 5 : cephadm [INF] Generating ssh key... 2023-07-10T16:54:55.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:55 smithi116 bash[23740]: audit 2023-07-10T16:54:54.535579+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' 2023-07-10T16:54:55.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:55 smithi116 bash[23740]: audit 2023-07-10T16:54:54.540859+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' 2023-07-10T16:54:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi6Cop+I/ebAqs/KzjXXdnP8HWSPvHNjyibwz1OoafKda7pF+UiYHodK6/uMwtGXnbsUuFq3lwtnfjtnWmLc29v4vBS103hy48+PwusV/w4mycFOwg3e26Y5ehEwL+C/iFXZPQ2xv0k+OfXnP1xR6HO1Iyem9LEzRctbGNvfwB8ZkwsWcBQoUPcADxFx//p91uoXpt3ggwduV8DYOfiydRmAlQkUpRulhDqa/Vo4FI7vfTNMdBraqNUqRfOYdzMzAaGPubVGH+Np49YlnQnYEXCc3xH0G8TCxUZuqPIkk1mBJ7tb+wSkT5u/h5FMltEerKq4fXTgnUHgTEY7YpWd9xuhE4Q5L+pnL3z7ZC9NrtJoBD+95uEETz6MldzYNCIVk5c7By/xyfw8VfQfMo2kDozjeyWls13vNn56Alwn7IVflLxfuj58Cx3mU0m9UhzrN/B3OC3XqXiAEeHjZuxkHNCNUV+JL3SqAmt+uK00rcnAfA1YK9sGE3SYUzvZs4OA8= ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:54:57.113 INFO:teuthology.orchestra.run.smithi116.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-10T16:54:57.113 INFO:teuthology.orchestra.run.smithi116.stderr:Adding key to root@localhost authorized_keys... 2023-07-10T16:54:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:Adding host smithi116... 2023-07-10T16:54:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 orch host add smithi116 2023-07-10T16:54:57.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:54:57 smithi116 bash[23740]: audit 2023-07-10T16:54:56.474469+0000 mgr.y (mgr.14116) 6 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:54:58.895 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:54:58 smithi116 bash[24013]: Warning: Permanently added 'smithi116,172.21.15.116' (ECDSA) to the list of known hosts. 2023-07-10T16:54:59.327 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: Added host 'smithi116' 2023-07-10T16:55:00.200 INFO:teuthology.orchestra.run.smithi116.stderr:Deploying unmanaged mon service... 2023-07-10T16:55:00.201 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 orch apply mon --unmanaged 2023-07-10T16:55:00.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:00 smithi116 bash[23740]: audit 2023-07-10T16:54:58.562872+0000 mgr.y (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi116", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:55:00.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:00 smithi116 bash[23740]: audit 2023-07-10T16:54:59.320052+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' 2023-07-10T16:55:00.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:00 smithi116 bash[23740]: cephadm 2023-07-10T16:54:59.320846+0000 mgr.y (mgr.14116) 8 : cephadm [INF] Added host smithi116 2023-07-10T16:55:00.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:00 smithi116 bash[23740]: audit 2023-07-10T16:54:59.321560+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:55:02.082 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: Scheduled mon update... 2023-07-10T16:55:02.989 INFO:teuthology.orchestra.run.smithi116.stderr:Deploying unmanaged mgr service... 2023-07-10T16:55:02.990 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 orch apply mgr --unmanaged 2023-07-10T16:55:03.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:03 smithi116 bash[23740]: audit 2023-07-10T16:55:02.069271+0000 mgr.y (mgr.14116) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:55:03.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:03 smithi116 bash[23740]: cephadm 2023-07-10T16:55:02.071007+0000 mgr.y (mgr.14116) 10 : cephadm [INF] Saving service mon spec with placement count:5 2023-07-10T16:55:03.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:03 smithi116 bash[23740]: audit 2023-07-10T16:55:02.074430+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' 2023-07-10T16:55:03.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:03 smithi116 bash[23740]: audit 2023-07-10T16:55:02.923701+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' 2023-07-10T16:55:04.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:04 smithi116 bash[23740]: audit 2023-07-10T16:55:03.179501+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' 2023-07-10T16:55:04.547 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-07-10T16:55:05.214 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config set mgr mgr/cephadm/container_init True --force 2023-07-10T16:55:05.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:05 smithi116 bash[23740]: audit 2023-07-10T16:55:04.533734+0000 mgr.y (mgr.14116) 11 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:55:05.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:05 smithi116 bash[23740]: cephadm 2023-07-10T16:55:04.534863+0000 mgr.y (mgr.14116) 12 : cephadm [INF] Saving service mgr spec with placement count:2 2023-07-10T16:55:05.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:05 smithi116 bash[23740]: audit 2023-07-10T16:55:04.540030+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' 2023-07-10T16:55:07.439 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config set mgr mgr/dashboard/ssl_server_port 8443 2023-07-10T16:55:07.472 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:07 smithi116 bash[23740]: audit 2023-07-10T16:55:06.213933+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.116:0/2979684645' entity='mgr.y' 2023-07-10T16:55:07.472 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:07 smithi116 bash[23740]: audit 2023-07-10T16:55:06.510237+0000 mon.a (mon.0) 62 : audit [INF] from='client.? 172.21.15.116:0/636646710' entity='client.admin' 2023-07-10T16:55:10.267 INFO:teuthology.orchestra.run.smithi116.stderr:Enabling the dashboard module... 2023-07-10T16:55:10.267 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 mgr module enable dashboard 2023-07-10T16:55:10.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:10 smithi116 bash[23740]: cluster 2023-07-10T16:55:09.343882+0000 mgr.y (mgr.14116) 13 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:55:11.110 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:10 smithi116 bash[23740]: audit 2023-07-10T16:55:09.581345+0000 mon.a (mon.0) 63 : audit [INF] from='client.? 172.21.15.116:0/771501025' entity='client.admin' 2023-07-10T16:55:12.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:12 smithi116 bash[23740]: cluster 2023-07-10T16:55:11.344284+0000 mgr.y (mgr.14116) 14 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:55:12.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:12 smithi116 bash[23740]: audit 2023-07-10T16:55:11.762279+0000 mon.a (mon.0) 64 : audit [INF] from='client.? 172.21.15.116:0/1777069032' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-10T16:55:12.896 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:55:12 smithi116 bash[24013]: ignoring --setuser ceph since I am not root 2023-07-10T16:55:12.896 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:55:12 smithi116 bash[24013]: ignoring --setgroup ceph since I am not root 2023-07-10T16:55:13.310 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 mgr stat 2023-07-10T16:55:13.900 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:13 smithi116 bash[23740]: audit 2023-07-10T16:55:12.589921+0000 mon.a (mon.0) 65 : audit [INF] from='client.? 172.21.15.116:0/1777069032' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-10T16:55:13.900 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:13 smithi116 bash[23740]: cluster 2023-07-10T16:55:12.590087+0000 mon.a (mon.0) 66 : cluster [DBG] mgrmap e9: y(active, since 23s) 2023-07-10T16:55:14.721 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: { 2023-07-10T16:55:14.721 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "epoch": 9, 2023-07-10T16:55:14.721 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "available": true, 2023-07-10T16:55:14.722 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "active_name": "y", 2023-07-10T16:55:14.722 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "num_standby": 0 2023-07-10T16:55:14.722 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: } 2023-07-10T16:55:15.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:14 smithi116 bash[23740]: audit 2023-07-10T16:55:14.713521+0000 mon.a (mon.0) 67 : audit [DBG] from='client.? 172.21.15.116:0/3064090925' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-10T16:55:15.362 INFO:teuthology.orchestra.run.smithi116.stderr:Waiting for the mgr to restart... 2023-07-10T16:55:15.362 INFO:teuthology.orchestra.run.smithi116.stderr:Waiting for mgr epoch 9... 2023-07-10T16:55:15.363 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 tell mgr mgr_status 2023-07-10T16:55:19.896 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:55:19 smithi116 bash[24013]: Warning: Permanently added 'smithi116,172.21.15.116' (ECDSA) to the list of known hosts. 2023-07-10T16:55:19.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:19 smithi116 bash[23740]: cluster 2023-07-10T16:55:18.480983+0000 mon.a (mon.0) 68 : cluster [INF] Active manager daemon y restarted 2023-07-10T16:55:19.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:19 smithi116 bash[23740]: cluster 2023-07-10T16:55:18.481762+0000 mon.a (mon.0) 69 : cluster [INF] Activating manager daemon y 2023-07-10T16:55:19.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:19 smithi116 bash[23740]: cluster 2023-07-10T16:55:18.484702+0000 mon.a (mon.0) 70 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-07-10T16:55:19.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:19 smithi116 bash[23740]: cluster 2023-07-10T16:55:19.342189+0000 mon.a (mon.0) 71 : cluster [DBG] mgrmap e10: y(active, starting, since 0.860554s) 2023-07-10T16:55:19.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:19 smithi116 bash[23740]: audit 2023-07-10T16:55:19.344967+0000 mon.a (mon.0) 72 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T16:55:19.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:19 smithi116 bash[23740]: audit 2023-07-10T16:55:19.347005+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-10T16:55:19.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:19 smithi116 bash[23740]: audit 2023-07-10T16:55:19.348487+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-10T16:55:19.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:19 smithi116 bash[23740]: audit 2023-07-10T16:55:19.348955+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-10T16:55:19.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:19 smithi116 bash[23740]: audit 2023-07-10T16:55:19.349378+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-10T16:55:19.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:19 smithi116 bash[23740]: cluster 2023-07-10T16:55:19.353826+0000 mon.a (mon.0) 77 : cluster [INF] Manager daemon y is now available 2023-07-10T16:55:20.370 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: { 2023-07-10T16:55:20.371 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2023-07-10T16:55:20.371 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: "initialized": true 2023-07-10T16:55:20.371 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: } 2023-07-10T16:55:20.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:20 smithi116 bash[23740]: audit 2023-07-10T16:55:19.507977+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:55:20.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:20 smithi116 bash[23740]: audit 2023-07-10T16:55:19.514296+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:55:20.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:20 smithi116 bash[23740]: audit 2023-07-10T16:55:19.555268+0000 mon.a (mon.0) 80 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T16:55:20.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:20 smithi116 bash[23740]: audit 2023-07-10T16:55:19.557257+0000 mon.a (mon.0) 81 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T16:55:20.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:20 smithi116 bash[23740]: cluster 2023-07-10T16:55:20.347601+0000 mon.a (mon.0) 82 : cluster [DBG] mgrmap e11: y(active, since 1.86597s) 2023-07-10T16:55:21.089 INFO:teuthology.orchestra.run.smithi116.stderr:mgr epoch 9 is available 2023-07-10T16:55:21.090 INFO:teuthology.orchestra.run.smithi116.stderr:Generating a dashboard self-signed certificate... 2023-07-10T16:55:21.090 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 dashboard create-self-signed-cert 2023-07-10T16:55:21.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:21 smithi116 bash[23740]: audit 2023-07-10T16:55:20.346703+0000 mgr.y (mgr.14144) 1 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-10T16:55:21.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:21 smithi116 bash[23740]: audit 2023-07-10T16:55:20.363381+0000 mgr.y (mgr.14144) 2 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-10T16:55:21.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:21 smithi116 bash[23740]: cluster 2023-07-10T16:55:21.347204+0000 mon.a (mon.0) 83 : cluster [DBG] mgrmap e12: y(active, since 2s) 2023-07-10T16:55:22.645 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: Self-signed certificate created 2023-07-10T16:55:23.308 INFO:teuthology.orchestra.run.smithi116.stderr:Creating initial admin user... 2023-07-10T16:55:23.309 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp2tzaf95h:/tmp/dashboard.pw:z quay.io/ceph/ceph:v16.2.0 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-07-10T16:55:23.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:23 smithi116 bash[23740]: audit 2023-07-10T16:55:22.458984+0000 mgr.y (mgr.14144) 3 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:55:23.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:23 smithi116 bash[23740]: audit 2023-07-10T16:55:22.629630+0000 mon.a (mon.0) 84 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:23.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:23 smithi116 bash[23740]: audit 2023-07-10T16:55:22.636706+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:25.006 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$u4KNWZh9Klcg/gvxDHz22.d3JQApHIH9gf30qCxwOoTx4S3uLfexe", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689008124, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-10T16:55:25.433 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:25 smithi116 bash[23740]: audit 2023-07-10T16:55:24.093167+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:25.434 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:25 smithi116 bash[23740]: audit 2023-07-10T16:55:24.998788+0000 mon.a (mon.0) 87 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:25.863 INFO:teuthology.orchestra.run.smithi116.stderr:Fetching dashboard port number... 2023-07-10T16:55:25.863 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi116 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzefxkes5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp12nvqx2z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config get mgr mgr/dashboard/ssl_server_port 2023-07-10T16:55:26.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:26 smithi116 bash[23740]: audit 2023-07-10T16:55:24.779402+0000 mgr.y (mgr.14144) 4 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:55:27.941 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/ceph: 8443 2023-07-10T16:55:28.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:28 smithi116 bash[23740]: cluster 2023-07-10T16:55:27.099911+0000 mon.a (mon.0) 88 : cluster [DBG] mgrmap e13: y(active, since 8s) 2023-07-10T16:55:28.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:28 smithi116 bash[23740]: audit 2023-07-10T16:55:27.933966+0000 mon.a (mon.0) 89 : audit [DBG] from='client.? 172.21.15.116:0/750829371' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-10T16:55:28.672 INFO:teuthology.orchestra.run.smithi116.stderr:firewalld does not appear to be present 2023-07-10T16:55:28.672 INFO:teuthology.orchestra.run.smithi116.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-10T16:55:28.677 INFO:teuthology.orchestra.run.smithi116.stderr:Ceph Dashboard is now available at: 2023-07-10T16:55:28.677 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-07-10T16:55:28.677 INFO:teuthology.orchestra.run.smithi116.stderr: URL: https://smithi116.front.sepia.ceph.com:8443/ 2023-07-10T16:55:28.678 INFO:teuthology.orchestra.run.smithi116.stderr: User: admin 2023-07-10T16:55:28.678 INFO:teuthology.orchestra.run.smithi116.stderr: Password: sx2r96ax27 2023-07-10T16:55:28.678 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-07-10T16:55:28.678 INFO:teuthology.orchestra.run.smithi116.stderr:You can access the Ceph CLI with: 2023-07-10T16:55:28.679 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-07-10T16:55:28.679 INFO:teuthology.orchestra.run.smithi116.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-10T16:55:28.679 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-07-10T16:55:28.679 INFO:teuthology.orchestra.run.smithi116.stderr:Please consider enabling telemetry to help improve Ceph: 2023-07-10T16:55:28.679 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-07-10T16:55:28.680 INFO:teuthology.orchestra.run.smithi116.stderr: ceph telemetry on 2023-07-10T16:55:28.680 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-07-10T16:55:28.680 INFO:teuthology.orchestra.run.smithi116.stderr:For more information see: 2023-07-10T16:55:28.680 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-07-10T16:55:28.681 INFO:teuthology.orchestra.run.smithi116.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-07-10T16:55:28.681 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-07-10T16:55:28.681 INFO:teuthology.orchestra.run.smithi116.stderr:Bootstrap complete. 2023-07-10T16:55:28.681 INFO:teuthology.orchestra.run.smithi116.stderr:Releasing lock 140519027428464 on /run/cephadm/3f83c2ec-1f42-11ee-9b31-001a4aab830c.lock 2023-07-10T16:55:28.681 INFO:teuthology.orchestra.run.smithi116.stderr:Lock 140519027428464 released on /run/cephadm/3f83c2ec-1f42-11ee-9b31-001a4aab830c.lock 2023-07-10T16:55:28.708 INFO:tasks.cephadm:Fetching config... 2023-07-10T16:55:28.708 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-07-10T16:55:28.709 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-10T16:55:28.713 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-10T16:55:28.713 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-07-10T16:55:28.714 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-10T16:55:28.722 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-10T16:55:28.723 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-07-10T16:55:28.724 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-10T16:55:28.735 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-10T16:55:28.735 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-07-10T16:55:28.735 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-10T16:55:28.782 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-10T16:55:28.783 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi6Cop+I/ebAqs/KzjXXdnP8HWSPvHNjyibwz1OoafKda7pF+UiYHodK6/uMwtGXnbsUuFq3lwtnfjtnWmLc29v4vBS103hy48+PwusV/w4mycFOwg3e26Y5ehEwL+C/iFXZPQ2xv0k+OfXnP1xR6HO1Iyem9LEzRctbGNvfwB8ZkwsWcBQoUPcADxFx//p91uoXpt3ggwduV8DYOfiydRmAlQkUpRulhDqa/Vo4FI7vfTNMdBraqNUqRfOYdzMzAaGPubVGH+Np49YlnQnYEXCc3xH0G8TCxUZuqPIkk1mBJ7tb+wSkT5u/h5FMltEerKq4fXTgnUHgTEY7YpWd9xuhE4Q5L+pnL3z7ZC9NrtJoBD+95uEETz6MldzYNCIVk5c7By/xyfw8VfQfMo2kDozjeyWls13vNn56Alwn7IVflLxfuj58Cx3mU0m9UhzrN/B3OC3XqXiAEeHjZuxkHNCNUV+JL3SqAmt+uK00rcnAfA1YK9sGE3SYUzvZs4OA8= ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-10T16:55:28.809 INFO:teuthology.orchestra.run.smithi116.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi6Cop+I/ebAqs/KzjXXdnP8HWSPvHNjyibwz1OoafKda7pF+UiYHodK6/uMwtGXnbsUuFq3lwtnfjtnWmLc29v4vBS103hy48+PwusV/w4mycFOwg3e26Y5ehEwL+C/iFXZPQ2xv0k+OfXnP1xR6HO1Iyem9LEzRctbGNvfwB8ZkwsWcBQoUPcADxFx//p91uoXpt3ggwduV8DYOfiydRmAlQkUpRulhDqa/Vo4FI7vfTNMdBraqNUqRfOYdzMzAaGPubVGH+Np49YlnQnYEXCc3xH0G8TCxUZuqPIkk1mBJ7tb+wSkT5u/h5FMltEerKq4fXTgnUHgTEY7YpWd9xuhE4Q5L+pnL3z7ZC9NrtJoBD+95uEETz6MldzYNCIVk5c7By/xyfw8VfQfMo2kDozjeyWls13vNn56Alwn7IVflLxfuj58Cx3mU0m9UhzrN/B3OC3XqXiAEeHjZuxkHNCNUV+JL3SqAmt+uK00rcnAfA1YK9sGE3SYUzvZs4OA8= ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:55:28.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi6Cop+I/ebAqs/KzjXXdnP8HWSPvHNjyibwz1OoafKda7pF+UiYHodK6/uMwtGXnbsUuFq3lwtnfjtnWmLc29v4vBS103hy48+PwusV/w4mycFOwg3e26Y5ehEwL+C/iFXZPQ2xv0k+OfXnP1xR6HO1Iyem9LEzRctbGNvfwB8ZkwsWcBQoUPcADxFx//p91uoXpt3ggwduV8DYOfiydRmAlQkUpRulhDqa/Vo4FI7vfTNMdBraqNUqRfOYdzMzAaGPubVGH+Np49YlnQnYEXCc3xH0G8TCxUZuqPIkk1mBJ7tb+wSkT5u/h5FMltEerKq4fXTgnUHgTEY7YpWd9xuhE4Q5L+pnL3z7ZC9NrtJoBD+95uEETz6MldzYNCIVk5c7By/xyfw8VfQfMo2kDozjeyWls13vNn56Alwn7IVflLxfuj58Cx3mU0m9UhzrN/B3OC3XqXiAEeHjZuxkHNCNUV+JL3SqAmt+uK00rcnAfA1YK9sGE3SYUzvZs4OA8= ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-10T16:55:28.842 INFO:teuthology.orchestra.run.smithi192.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi6Cop+I/ebAqs/KzjXXdnP8HWSPvHNjyibwz1OoafKda7pF+UiYHodK6/uMwtGXnbsUuFq3lwtnfjtnWmLc29v4vBS103hy48+PwusV/w4mycFOwg3e26Y5ehEwL+C/iFXZPQ2xv0k+OfXnP1xR6HO1Iyem9LEzRctbGNvfwB8ZkwsWcBQoUPcADxFx//p91uoXpt3ggwduV8DYOfiydRmAlQkUpRulhDqa/Vo4FI7vfTNMdBraqNUqRfOYdzMzAaGPubVGH+Np49YlnQnYEXCc3xH0G8TCxUZuqPIkk1mBJ7tb+wSkT5u/h5FMltEerKq4fXTgnUHgTEY7YpWd9xuhE4Q5L+pnL3z7ZC9NrtJoBD+95uEETz6MldzYNCIVk5c7By/xyfw8VfQfMo2kDozjeyWls13vNn56Alwn7IVflLxfuj58Cx3mU0m9UhzrN/B3OC3XqXiAEeHjZuxkHNCNUV+JL3SqAmt+uK00rcnAfA1YK9sGE3SYUzvZs4OA8= ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:55:28.853 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi192 2023-07-10T16:55:28.854 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-10T16:55:28.854 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.conf 2023-07-10T16:55:28.907 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-10T16:55:28.907 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-10T16:55:28.915 INFO:tasks.cephadm:Adding host smithi192 to orchestrator... 2023-07-10T16:55:28.916 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph orch host add smithi192 2023-07-10T16:55:33.146 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 16:55:32 smithi116 bash[24013]: Warning: Permanently added 'smithi192,172.21.15.192' (ECDSA) to the list of known hosts. 2023-07-10T16:55:33.662 INFO:teuthology.orchestra.run.smithi116.stdout:Added host 'smithi192' 2023-07-10T16:55:34.352 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph orch host ls --format=json 2023-07-10T16:55:34.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:34 smithi116 bash[23740]: audit 2023-07-10T16:55:32.849196+0000 mgr.y (mgr.14144) 5 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:55:34.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:34 smithi116 bash[23740]: audit 2023-07-10T16:55:33.654321+0000 mon.a (mon.0) 90 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:35.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:35 smithi116 bash[23740]: cephadm 2023-07-10T16:55:33.655420+0000 mgr.y (mgr.14144) 6 : cephadm [INF] Added host smithi192 2023-07-10T16:55:37.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:37 smithi116 bash[23740]: audit 2023-07-10T16:55:36.124826+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:37.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:37 smithi116 bash[23740]: audit 2023-07-10T16:55:36.563520+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:37.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:37 smithi116 bash[23740]: audit 2023-07-10T16:55:36.566287+0000 mon.a (mon.0) 93 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:55:37.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:37 smithi116 bash[23740]: audit 2023-07-10T16:55:36.819618+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:38.102 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-07-10T16:55:38.102 INFO:teuthology.orchestra.run.smithi116.stdout:[{"addr": "smithi116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "smithi192", "hostname": "smithi192", "labels": [], "status": ""}] 2023-07-10T16:55:38.747 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-10T16:55:38.748 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph osd crush tunables default 2023-07-10T16:55:38.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:38 smithi116 bash[23740]: audit 2023-07-10T16:55:37.534464+0000 mon.a (mon.0) 95 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:39.812 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:39 smithi116 bash[23740]: audit 2023-07-10T16:55:38.094569+0000 mgr.y (mgr.14144) 7 : audit [DBG] from='client.14162 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-10T16:55:40.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:40 smithi116 bash[23740]: cluster 2023-07-10T16:55:39.350335+0000 mgr.y (mgr.14144) 8 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:55:42.559 INFO:teuthology.orchestra.run.smithi116.stderr:adjusted tunables profile to default 2023-07-10T16:55:42.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:42 smithi116 bash[23740]: cluster 2023-07-10T16:55:41.350775+0000 mgr.y (mgr.14144) 9 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:55:42.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:42 smithi116 bash[23740]: audit 2023-07-10T16:55:41.708664+0000 mon.a (mon.0) 96 : audit [INF] from='client.? 172.21.15.116:0/285915187' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-10T16:55:42.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:42 smithi116 bash[23740]: audit 2023-07-10T16:55:42.029738+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:42.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:42 smithi116 bash[23740]: audit 2023-07-10T16:55:42.301972+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:42.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:42 smithi116 bash[23740]: audit 2023-07-10T16:55:42.307434+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:43.251 INFO:tasks.cephadm:Adding mon.a on smithi116 2023-07-10T16:55:43.252 INFO:tasks.cephadm:Adding mon.c on smithi116 2023-07-10T16:55:43.252 INFO:tasks.cephadm:Adding mon.b on smithi192 2023-07-10T16:55:43.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph orch apply mon '3;smithi116:172.21.15.116=a;smithi116:[v2:172.21.15.116:3301,v1:172.21.15.116:6790]=c;smithi192:172.21.15.192=b' 2023-07-10T16:55:43.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:43 smithi116 bash[23740]: audit 2023-07-10T16:55:42.544800+0000 mon.a (mon.0) 100 : audit [INF] from='client.? 172.21.15.116:0/285915187' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-10T16:55:43.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:43 smithi116 bash[23740]: cluster 2023-07-10T16:55:42.544989+0000 mon.a (mon.0) 101 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-07-10T16:55:44.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:44 smithi116 bash[23740]: cluster 2023-07-10T16:55:43.351284+0000 mgr.y (mgr.14144) 10 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:55:46.012 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mon update... 2023-07-10T16:55:46.601 DEBUG:teuthology.orchestra.run.smithi116:mon.c> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.c.service 2023-07-10T16:55:46.604 DEBUG:teuthology.orchestra.run.smithi192:mon.b> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.b.service 2023-07-10T16:55:46.607 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-10T16:55:46.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph mon dump -f json 2023-07-10T16:55:46.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:46 smithi116 bash[23740]: cluster 2023-07-10T16:55:45.351926+0000 mgr.y (mgr.14144) 11 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:55:46.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:46 smithi116 bash[23740]: audit 2023-07-10T16:55:46.004749+0000 mon.a (mon.0) 102 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:46.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:46 smithi116 bash[23740]: audit 2023-07-10T16:55:46.006133+0000 mon.a (mon.0) 103 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:55:46.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:46 smithi116 bash[23740]: audit 2023-07-10T16:55:46.013827+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:46.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:46 smithi116 bash[23740]: audit 2023-07-10T16:55:46.016137+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T16:55:46.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:46 smithi116 bash[23740]: audit 2023-07-10T16:55:46.017782+0000 mon.a (mon.0) 106 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:55:47.163 INFO:journalctl@ceph.mon.b.smithi192.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T16:55:47.439 INFO:journalctl@ceph.mon.c.smithi116.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T16:55:47.729 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:47 smithi116 bash[23740]: audit 2023-07-10T16:55:45.994157+0000 mgr.y (mgr.14144) 12 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi116:172.21.15.116=a;smithi116:[v2:172.21.15.116:3301,v1:172.21.15.116:6790]=c;smithi192:172.21.15.192=b", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:55:47.730 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:47 smithi116 bash[23740]: cephadm 2023-07-10T16:55:45.997492+0000 mgr.y (mgr.14144) 13 : cephadm [INF] Saving service mon spec with placement smithi116:172.21.15.116=a;smithi116:[v2:172.21.15.116:3301,v1:172.21.15.116:6790]=c;smithi192:172.21.15.192=b;count:3 2023-07-10T16:55:47.730 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:47 smithi116 bash[23740]: cephadm 2023-07-10T16:55:46.019542+0000 mgr.y (mgr.14144) 14 : cephadm [INF] Deploying daemon mon.c on smithi116 2023-07-10T16:55:48.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:48 smithi116 bash[23740]: cluster 2023-07-10T16:55:47.352404+0000 mgr.y (mgr.14144) 15 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:55:49.469 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-10T16:55:49.470 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"3f83c2ec-1f42-11ee-9b31-001a4aab830c","modified":"2023-07-10T16:54:17.174185Z","created":"2023-07-10T16:54:17.174185Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-10T16:55:49.472 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2023-07-10T16:55:49.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:49 smithi116 bash[23740]: audit 2023-07-10T16:55:49.462673+0000 mon.a (mon.0) 107 : audit [DBG] from='client.? 172.21.15.192:0/1042009387' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-10T16:55:50.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:50 smithi116 bash[23740]: cluster 2023-07-10T16:55:49.352885+0000 mgr.y (mgr.14144) 16 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:55:50.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:50 smithi116 bash[23740]: audit 2023-07-10T16:55:50.348177+0000 mon.a (mon.0) 108 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:50.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:50 smithi116 bash[23740]: audit 2023-07-10T16:55:50.349041+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T16:55:50.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:50 smithi116 bash[23740]: audit 2023-07-10T16:55:50.350010+0000 mon.a (mon.0) 110 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:55:51.061 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-10T16:55:51.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph mon dump -f json 2023-07-10T16:55:51.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:51 smithi116 bash[28351]: debug 2023-07-10T16:55:51.452+0000 7f25c3267700 10 mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-10T16:55:56.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: cluster 2023-07-10T16:55:51.353392+0000 mgr.y (mgr.14144) 18 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:55:56.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: audit 2023-07-10T16:55:51.464686+0000 mon.a (mon.0) 112 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T16:55:56.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: cluster 2023-07-10T16:55:51.465011+0000 mon.a (mon.0) 113 : cluster [INF] mon.a calling monitor election 2023-07-10T16:55:56.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: audit 2023-07-10T16:55:51.467273+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T16:55:56.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: audit 2023-07-10T16:55:52.460540+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T16:55:56.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: cluster 2023-07-10T16:55:53.353837+0000 mgr.y (mgr.14144) 19 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:55:56.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: audit 2023-07-10T16:55:53.460738+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T16:55:56.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: cluster 2023-07-10T16:55:53.463093+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2023-07-10T16:55:56.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: audit 2023-07-10T16:55:54.460761+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T16:55:56.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: cluster 2023-07-10T16:55:55.354242+0000 mgr.y (mgr.14144) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:55:56.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: audit 2023-07-10T16:55:55.460988+0000 mon.a (mon.0) 118 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T16:55:56.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: audit 2023-07-10T16:55:56.461040+0000 mon.a (mon.0) 119 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T16:55:56.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: cluster 2023-07-10T16:55:56.488821+0000 mon.a (mon.0) 120 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-10T16:55:56.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: cluster 2023-07-10T16:55:56.495924+0000 mon.a (mon.0) 121 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0]} 2023-07-10T16:55:56.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: cluster 2023-07-10T16:55:56.496033+0000 mon.a (mon.0) 122 : cluster [DBG] fsmap 2023-07-10T16:55:56.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: cluster 2023-07-10T16:55:56.496083+0000 mon.a (mon.0) 123 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-07-10T16:55:56.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: cluster 2023-07-10T16:55:56.496400+0000 mon.a (mon.0) 124 : cluster [DBG] mgrmap e13: y(active, since 38s) 2023-07-10T16:55:56.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: cluster 2023-07-10T16:55:56.503423+0000 mon.a (mon.0) 125 : cluster [INF] overall HEALTH_OK 2023-07-10T16:55:56.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: audit 2023-07-10T16:55:56.507988+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:56.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[23740]: audit 2023-07-10T16:55:56.509059+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:55:56.904 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: cluster 2023-07-10T16:55:51.353392+0000 mgr.y (mgr.14144) 18 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:55:56.905 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: audit 2023-07-10T16:55:51.464686+0000 mon.a (mon.0) 112 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T16:55:56.905 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: cluster 2023-07-10T16:55:51.465011+0000 mon.a (mon.0) 113 : cluster [INF] mon.a calling monitor election 2023-07-10T16:55:56.905 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: audit 2023-07-10T16:55:51.467273+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T16:55:56.905 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: audit 2023-07-10T16:55:52.460540+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T16:55:56.905 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: cluster 2023-07-10T16:55:53.353837+0000 mgr.y (mgr.14144) 19 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:55:56.905 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: audit 2023-07-10T16:55:53.460738+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T16:55:56.906 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: cluster 2023-07-10T16:55:53.463093+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2023-07-10T16:55:56.906 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: audit 2023-07-10T16:55:54.460761+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T16:55:56.906 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: cluster 2023-07-10T16:55:55.354242+0000 mgr.y (mgr.14144) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:55:56.906 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: audit 2023-07-10T16:55:55.460988+0000 mon.a (mon.0) 118 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T16:55:56.906 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: audit 2023-07-10T16:55:56.461040+0000 mon.a (mon.0) 119 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T16:55:56.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: cluster 2023-07-10T16:55:56.488821+0000 mon.a (mon.0) 120 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-10T16:55:56.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: cluster 2023-07-10T16:55:56.495924+0000 mon.a (mon.0) 121 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0]} 2023-07-10T16:55:56.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: cluster 2023-07-10T16:55:56.496033+0000 mon.a (mon.0) 122 : cluster [DBG] fsmap 2023-07-10T16:55:56.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: cluster 2023-07-10T16:55:56.496083+0000 mon.a (mon.0) 123 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-07-10T16:55:56.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: cluster 2023-07-10T16:55:56.496400+0000 mon.a (mon.0) 124 : cluster [DBG] mgrmap e13: y(active, since 38s) 2023-07-10T16:55:56.908 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: cluster 2023-07-10T16:55:56.503423+0000 mon.a (mon.0) 125 : cluster [INF] overall HEALTH_OK 2023-07-10T16:55:56.909 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: audit 2023-07-10T16:55:56.507988+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:55:56.909 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:55:56 smithi116 bash[28351]: audit 2023-07-10T16:55:56.509059+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:55:57.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:55:56 smithi192 bash[23730]: debug 2023-07-10T16:55:56.945+0000 7f172827a700 1 mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2023-07-10T16:56:02.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: audit 2023-07-10T16:55:56.959470+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T16:56:02.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: cluster 2023-07-10T16:55:56.959919+0000 mon.a (mon.0) 130 : cluster [INF] mon.a calling monitor election 2023-07-10T16:56:02.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: cluster 2023-07-10T16:55:56.960327+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2023-07-10T16:56:02.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: audit 2023-07-10T16:55:56.962197+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:02.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: audit 2023-07-10T16:55:56.962492+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T16:56:02.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: cluster 2023-07-10T16:55:57.354662+0000 mgr.y (mgr.14144) 21 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:02.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: audit 2023-07-10T16:55:57.952287+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:02.400 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: audit 2023-07-10T16:55:58.952170+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:02.400 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: cluster 2023-07-10T16:55:59.355084+0000 mgr.y (mgr.14144) 22 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:02.400 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: audit 2023-07-10T16:55:59.952478+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:02.401 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: audit 2023-07-10T16:56:00.952685+0000 mon.a (mon.0) 136 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:02.401 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: cluster 2023-07-10T16:56:01.355477+0000 mgr.y (mgr.14144) 23 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:02.401 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: audit 2023-07-10T16:56:01.952503+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:02.401 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: cluster 2023-07-10T16:56:01.962195+0000 mon.a (mon.0) 138 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-10T16:56:02.402 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: cluster 2023-07-10T16:56:01.969073+0000 mon.a (mon.0) 139 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0]} 2023-07-10T16:56:02.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: cluster 2023-07-10T16:56:01.969180+0000 mon.a (mon.0) 140 : cluster [DBG] fsmap 2023-07-10T16:56:02.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: cluster 2023-07-10T16:56:01.969241+0000 mon.a (mon.0) 141 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-07-10T16:56:02.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: cluster 2023-07-10T16:56:01.969566+0000 mon.a (mon.0) 142 : cluster [DBG] mgrmap e13: y(active, since 43s) 2023-07-10T16:56:02.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: cluster 2023-07-10T16:56:01.969917+0000 mon.a (mon.0) 143 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-10T16:56:02.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: cluster 2023-07-10T16:56:01.977371+0000 mon.a (mon.0) 144 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-10T16:56:02.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: cluster 2023-07-10T16:56:01.977419+0000 mon.a (mon.0) 145 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-10T16:56:02.407 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: audit 2023-07-10T16:55:56.959470+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T16:56:02.407 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: cluster 2023-07-10T16:55:56.959919+0000 mon.a (mon.0) 130 : cluster [INF] mon.a calling monitor election 2023-07-10T16:56:02.407 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: cluster 2023-07-10T16:55:56.960327+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2023-07-10T16:56:02.407 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: audit 2023-07-10T16:55:56.962197+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:02.408 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: audit 2023-07-10T16:55:56.962492+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T16:56:02.408 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: cluster 2023-07-10T16:55:57.354662+0000 mgr.y (mgr.14144) 21 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:02.408 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: audit 2023-07-10T16:55:57.952287+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:02.409 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: audit 2023-07-10T16:55:58.952170+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:02.409 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: cluster 2023-07-10T16:55:59.355084+0000 mgr.y (mgr.14144) 22 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:02.409 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: audit 2023-07-10T16:55:59.952478+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:02.410 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: audit 2023-07-10T16:56:00.952685+0000 mon.a (mon.0) 136 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:02.410 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: cluster 2023-07-10T16:56:01.355477+0000 mgr.y (mgr.14144) 23 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:02.410 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: audit 2023-07-10T16:56:01.952503+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:02.411 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: cluster 2023-07-10T16:56:01.962195+0000 mon.a (mon.0) 138 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-10T16:56:02.411 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: cluster 2023-07-10T16:56:01.969073+0000 mon.a (mon.0) 139 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0]} 2023-07-10T16:56:02.412 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: cluster 2023-07-10T16:56:01.969180+0000 mon.a (mon.0) 140 : cluster [DBG] fsmap 2023-07-10T16:56:02.412 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: cluster 2023-07-10T16:56:01.969241+0000 mon.a (mon.0) 141 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-07-10T16:56:02.412 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: cluster 2023-07-10T16:56:01.969566+0000 mon.a (mon.0) 142 : cluster [DBG] mgrmap e13: y(active, since 43s) 2023-07-10T16:56:02.413 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: cluster 2023-07-10T16:56:01.969917+0000 mon.a (mon.0) 143 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-10T16:56:02.413 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: cluster 2023-07-10T16:56:01.977371+0000 mon.a (mon.0) 144 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-10T16:56:02.413 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: cluster 2023-07-10T16:56:01.977419+0000 mon.a (mon.0) 145 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-10T16:56:02.415 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: cluster 2023-07-10T16:56:01.977483+0000 mon.a (mon.0) 146 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] is down (out of quorum) 2023-07-10T16:56:02.415 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: audit 2023-07-10T16:56:01.983679+0000 mon.a (mon.0) 147 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:02.415 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: audit 2023-07-10T16:56:01.989151+0000 mon.a (mon.0) 148 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:02.416 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: audit 2023-07-10T16:56:01.994855+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:02.416 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: audit 2023-07-10T16:56:01.997070+0000 mon.a (mon.0) 150 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T16:56:02.416 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: audit 2023-07-10T16:56:01.998559+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T16:56:02.416 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[23740]: audit 2023-07-10T16:56:01.999901+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:02.417 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: cluster 2023-07-10T16:56:01.977483+0000 mon.a (mon.0) 146 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] is down (out of quorum) 2023-07-10T16:56:02.417 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: audit 2023-07-10T16:56:01.983679+0000 mon.a (mon.0) 147 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:02.417 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: audit 2023-07-10T16:56:01.989151+0000 mon.a (mon.0) 148 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:02.418 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: audit 2023-07-10T16:56:01.994855+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:02.418 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: audit 2023-07-10T16:56:01.997070+0000 mon.a (mon.0) 150 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T16:56:02.418 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: audit 2023-07-10T16:56:01.998559+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T16:56:02.418 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:02 smithi116 bash[28351]: audit 2023-07-10T16:56:01.999901+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:02.631 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-07-10T16:56:02.631 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":3,"fsid":"3f83c2ec-1f42-11ee-9b31-001a4aab830c","modified":"2023-07-10T16:55:56.952736Z","created":"2023-07-10T16:54:17.174185Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-10T16:56:02.634 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 3 2023-07-10T16:56:03.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:03 smithi116 bash[23740]: cephadm 2023-07-10T16:56:01.996597+0000 mgr.y (mgr.14144) 24 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2023-07-10T16:56:03.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:03 smithi116 bash[23740]: cephadm 2023-07-10T16:56:02.001021+0000 mgr.y (mgr.14144) 25 : cephadm [INF] Reconfiguring daemon mon.a on smithi116 2023-07-10T16:56:03.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:03 smithi116 bash[23740]: audit 2023-07-10T16:56:02.623969+0000 mon.a (mon.0) 153 : audit [DBG] from='client.? 172.21.15.192:0/3940590324' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-10T16:56:03.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:03 smithi116 bash[23740]: audit 2023-07-10T16:56:02.952865+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:03.320 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:03 smithi116 bash[28351]: cephadm 2023-07-10T16:56:01.996597+0000 mgr.y (mgr.14144) 24 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2023-07-10T16:56:03.321 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:03 smithi116 bash[28351]: cephadm 2023-07-10T16:56:02.001021+0000 mgr.y (mgr.14144) 25 : cephadm [INF] Reconfiguring daemon mon.a on smithi116 2023-07-10T16:56:03.321 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:03 smithi116 bash[28351]: audit 2023-07-10T16:56:02.623969+0000 mon.a (mon.0) 153 : audit [DBG] from='client.? 172.21.15.192:0/3940590324' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-10T16:56:03.321 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:03 smithi116 bash[28351]: audit 2023-07-10T16:56:02.952865+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:03.425 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-10T16:56:03.425 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph config generate-minimal-conf 2023-07-10T16:56:04.356 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:04 smithi192 bash[23730]: cephadm 2023-07-10T16:56:01.996597+0000 mgr.y (mgr.14144) 24 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2023-07-10T16:56:04.356 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:04 smithi192 bash[23730]: cephadm 2023-07-10T16:56:02.001021+0000 mgr.y (mgr.14144) 25 : cephadm [INF] Reconfiguring daemon mon.a on smithi116 2023-07-10T16:56:04.356 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:04 smithi192 bash[23730]: audit 2023-07-10T16:56:02.623969+0000 mon.a (mon.0) 153 : audit [DBG] from='client.? 172.21.15.192:0/3940590324' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-10T16:56:04.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:04 smithi192 bash[23730]: audit 2023-07-10T16:56:02.952865+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:04.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:04 smithi192 bash[23730]: cluster 2023-07-10T16:55:58.960500+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2023-07-10T16:56:04.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:04 smithi192 bash[23730]: cluster 2023-07-10T16:56:03.974942+0000 mon.c (mon.1) 3 : cluster [INF] mon.c calling monitor election 2023-07-10T16:56:04.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:04 smithi192 bash[23730]: cluster 2023-07-10T16:56:03.974967+0000 mon.a (mon.0) 156 : cluster [INF] mon.a calling monitor election 2023-07-10T16:56:04.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:04 smithi192 bash[23730]: cluster 2023-07-10T16:56:03.979650+0000 mon.b (mon.2) 2 : cluster [INF] mon.b calling monitor election 2023-07-10T16:56:04.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:04 smithi192 bash[23730]: cluster 2023-07-10T16:56:03.995509+0000 mon.a (mon.0) 157 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-10T16:56:04.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:04 smithi192 bash[23730]: cluster 2023-07-10T16:56:04.002270+0000 mon.a (mon.0) 158 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0]} 2023-07-10T16:56:04.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:04 smithi192 bash[23730]: cluster 2023-07-10T16:56:04.002391+0000 mon.a (mon.0) 159 : cluster [DBG] fsmap 2023-07-10T16:56:04.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:04 smithi192 bash[23730]: cluster 2023-07-10T16:56:04.002450+0000 mon.a (mon.0) 160 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-07-10T16:56:04.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:04 smithi192 bash[23730]: cluster 2023-07-10T16:56:04.002776+0000 mon.a (mon.0) 161 : cluster [DBG] mgrmap e13: y(active, since 45s) 2023-07-10T16:56:04.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:04 smithi192 bash[23730]: cluster 2023-07-10T16:56:04.003097+0000 mon.a (mon.0) 162 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-10T16:56:04.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:04 smithi192 bash[23730]: cluster 2023-07-10T16:56:04.003146+0000 mon.a (mon.0) 163 : cluster [INF] Cluster is now healthy 2023-07-10T16:56:04.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:04 smithi192 bash[23730]: cluster 2023-07-10T16:56:04.010247+0000 mon.a (mon.0) 164 : cluster [INF] overall HEALTH_OK 2023-07-10T16:56:04.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[23740]: cluster 2023-07-10T16:55:58.960500+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2023-07-10T16:56:04.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[23740]: cluster 2023-07-10T16:56:03.974942+0000 mon.c (mon.1) 3 : cluster [INF] mon.c calling monitor election 2023-07-10T16:56:04.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[23740]: cluster 2023-07-10T16:56:03.974967+0000 mon.a (mon.0) 156 : cluster [INF] mon.a calling monitor election 2023-07-10T16:56:04.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[23740]: cluster 2023-07-10T16:56:03.979650+0000 mon.b (mon.2) 2 : cluster [INF] mon.b calling monitor election 2023-07-10T16:56:04.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[23740]: cluster 2023-07-10T16:56:03.995509+0000 mon.a (mon.0) 157 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-10T16:56:04.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[23740]: cluster 2023-07-10T16:56:04.002270+0000 mon.a (mon.0) 158 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0]} 2023-07-10T16:56:04.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[23740]: cluster 2023-07-10T16:56:04.002391+0000 mon.a (mon.0) 159 : cluster [DBG] fsmap 2023-07-10T16:56:04.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[23740]: cluster 2023-07-10T16:56:04.002450+0000 mon.a (mon.0) 160 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-07-10T16:56:04.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[23740]: cluster 2023-07-10T16:56:04.002776+0000 mon.a (mon.0) 161 : cluster [DBG] mgrmap e13: y(active, since 45s) 2023-07-10T16:56:04.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[23740]: cluster 2023-07-10T16:56:04.003097+0000 mon.a (mon.0) 162 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-10T16:56:04.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[23740]: cluster 2023-07-10T16:56:04.003146+0000 mon.a (mon.0) 163 : cluster [INF] Cluster is now healthy 2023-07-10T16:56:04.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[23740]: cluster 2023-07-10T16:56:04.010247+0000 mon.a (mon.0) 164 : cluster [INF] overall HEALTH_OK 2023-07-10T16:56:04.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[28351]: cluster 2023-07-10T16:55:58.960500+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2023-07-10T16:56:04.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[28351]: cluster 2023-07-10T16:56:03.974942+0000 mon.c (mon.1) 3 : cluster [INF] mon.c calling monitor election 2023-07-10T16:56:04.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[28351]: cluster 2023-07-10T16:56:03.974967+0000 mon.a (mon.0) 156 : cluster [INF] mon.a calling monitor election 2023-07-10T16:56:04.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[28351]: cluster 2023-07-10T16:56:03.979650+0000 mon.b (mon.2) 2 : cluster [INF] mon.b calling monitor election 2023-07-10T16:56:04.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[28351]: cluster 2023-07-10T16:56:03.995509+0000 mon.a (mon.0) 157 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-10T16:56:04.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[28351]: cluster 2023-07-10T16:56:04.002270+0000 mon.a (mon.0) 158 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0]} 2023-07-10T16:56:04.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[28351]: cluster 2023-07-10T16:56:04.002391+0000 mon.a (mon.0) 159 : cluster [DBG] fsmap 2023-07-10T16:56:04.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[28351]: cluster 2023-07-10T16:56:04.002450+0000 mon.a (mon.0) 160 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-07-10T16:56:04.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[28351]: cluster 2023-07-10T16:56:04.002776+0000 mon.a (mon.0) 161 : cluster [DBG] mgrmap e13: y(active, since 45s) 2023-07-10T16:56:04.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[28351]: cluster 2023-07-10T16:56:04.003097+0000 mon.a (mon.0) 162 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-10T16:56:04.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[28351]: cluster 2023-07-10T16:56:04.003146+0000 mon.a (mon.0) 163 : cluster [INF] Cluster is now healthy 2023-07-10T16:56:04.653 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:04 smithi116 bash[28351]: cluster 2023-07-10T16:56:04.010247+0000 mon.a (mon.0) 164 : cluster [INF] overall HEALTH_OK 2023-07-10T16:56:05.634 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:05 smithi192 bash[23730]: audit 2023-07-10T16:56:04.362537+0000 mon.a (mon.0) 165 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:05.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:05 smithi192 bash[23730]: cephadm 2023-07-10T16:56:04.363134+0000 mgr.y (mgr.14144) 27 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2023-07-10T16:56:05.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:05 smithi192 bash[23730]: audit 2023-07-10T16:56:04.363476+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T16:56:05.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:05 smithi192 bash[23730]: audit 2023-07-10T16:56:04.364439+0000 mon.a (mon.0) 167 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T16:56:05.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:05 smithi192 bash[23730]: audit 2023-07-10T16:56:04.365484+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:05.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:05 smithi192 bash[23730]: cephadm 2023-07-10T16:56:04.366418+0000 mgr.y (mgr.14144) 28 : cephadm [INF] Reconfiguring daemon mon.c on smithi116 2023-07-10T16:56:05.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:05 smithi192 bash[23730]: audit 2023-07-10T16:56:04.953157+0000 mon.a (mon.0) 169 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:05.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:05 smithi116 bash[23740]: audit 2023-07-10T16:56:04.362537+0000 mon.a (mon.0) 165 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:05.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:05 smithi116 bash[23740]: cephadm 2023-07-10T16:56:04.363134+0000 mgr.y (mgr.14144) 27 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2023-07-10T16:56:05.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:05 smithi116 bash[23740]: audit 2023-07-10T16:56:04.363476+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T16:56:05.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:05 smithi116 bash[23740]: audit 2023-07-10T16:56:04.364439+0000 mon.a (mon.0) 167 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T16:56:05.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:05 smithi116 bash[23740]: audit 2023-07-10T16:56:04.365484+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:05.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:05 smithi116 bash[23740]: cephadm 2023-07-10T16:56:04.366418+0000 mgr.y (mgr.14144) 28 : cephadm [INF] Reconfiguring daemon mon.c on smithi116 2023-07-10T16:56:05.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:05 smithi116 bash[23740]: audit 2023-07-10T16:56:04.953157+0000 mon.a (mon.0) 169 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:05.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:05 smithi116 bash[28351]: audit 2023-07-10T16:56:04.362537+0000 mon.a (mon.0) 165 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:05.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:05 smithi116 bash[28351]: cephadm 2023-07-10T16:56:04.363134+0000 mgr.y (mgr.14144) 27 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2023-07-10T16:56:05.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:05 smithi116 bash[28351]: audit 2023-07-10T16:56:04.363476+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T16:56:05.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:05 smithi116 bash[28351]: audit 2023-07-10T16:56:04.364439+0000 mon.a (mon.0) 167 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T16:56:05.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:05 smithi116 bash[28351]: audit 2023-07-10T16:56:04.365484+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:05.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:05 smithi116 bash[28351]: cephadm 2023-07-10T16:56:04.366418+0000 mgr.y (mgr.14144) 28 : cephadm [INF] Reconfiguring daemon mon.c on smithi116 2023-07-10T16:56:05.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:05 smithi116 bash[28351]: audit 2023-07-10T16:56:04.953157+0000 mon.a (mon.0) 169 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T16:56:06.634 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:06 smithi192 bash[23730]: cluster 2023-07-10T16:56:05.356275+0000 mgr.y (mgr.14144) 29 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:06.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:06 smithi116 bash[23740]: cluster 2023-07-10T16:56:05.356275+0000 mgr.y (mgr.14144) 29 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:06.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:06 smithi116 bash[28351]: cluster 2023-07-10T16:56:05.356275+0000 mgr.y (mgr.14144) 29 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:08.175 INFO:teuthology.orchestra.run.smithi116.stdout:# minimal ceph.conf for 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:56:08.176 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2023-07-10T16:56:08.176 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T16:56:08.176 INFO:teuthology.orchestra.run.smithi116.stdout: mon_host = [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2023-07-10T16:56:08.862 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-10T16:56:08.863 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-07-10T16:56:08.863 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.conf 2023-07-10T16:56:08.873 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-07-10T16:56:08.874 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-10T16:56:08.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:08 smithi116 bash[23740]: cluster 2023-07-10T16:56:07.356739+0000 mgr.y (mgr.14144) 30 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:08.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:08 smithi116 bash[23740]: audit 2023-07-10T16:56:07.622938+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:08.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:08 smithi116 bash[23740]: cephadm 2023-07-10T16:56:07.624042+0000 mgr.y (mgr.14144) 31 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2023-07-10T16:56:08.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:08 smithi116 bash[23740]: audit 2023-07-10T16:56:07.624540+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T16:56:08.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:08 smithi116 bash[23740]: audit 2023-07-10T16:56:07.626031+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T16:56:08.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:08 smithi116 bash[23740]: audit 2023-07-10T16:56:07.627558+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:08.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:08 smithi116 bash[23740]: cephadm 2023-07-10T16:56:07.628892+0000 mgr.y (mgr.14144) 32 : cephadm [INF] Reconfiguring daemon mon.b on smithi192 2023-07-10T16:56:08.884 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:08 smithi116 bash[23740]: audit 2023-07-10T16:56:08.168299+0000 mon.a (mon.0) 174 : audit [DBG] from='client.? 172.21.15.116:0/3832081129' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:08.884 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:08 smithi116 bash[28351]: cluster 2023-07-10T16:56:07.356739+0000 mgr.y (mgr.14144) 30 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:08.884 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:08 smithi116 bash[28351]: audit 2023-07-10T16:56:07.622938+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:08.884 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:08 smithi116 bash[28351]: cephadm 2023-07-10T16:56:07.624042+0000 mgr.y (mgr.14144) 31 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2023-07-10T16:56:08.884 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:08 smithi116 bash[28351]: audit 2023-07-10T16:56:07.624540+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T16:56:08.885 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:08 smithi116 bash[28351]: audit 2023-07-10T16:56:07.626031+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T16:56:08.885 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:08 smithi116 bash[28351]: audit 2023-07-10T16:56:07.627558+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:08.885 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:08 smithi116 bash[28351]: cephadm 2023-07-10T16:56:07.628892+0000 mgr.y (mgr.14144) 32 : cephadm [INF] Reconfiguring daemon mon.b on smithi192 2023-07-10T16:56:08.885 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:08 smithi116 bash[28351]: audit 2023-07-10T16:56:08.168299+0000 mon.a (mon.0) 174 : audit [DBG] from='client.? 172.21.15.116:0/3832081129' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:08.886 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-10T16:56:08.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.conf 2023-07-10T16:56:08.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:08 smithi192 bash[23730]: cluster 2023-07-10T16:56:07.356739+0000 mgr.y (mgr.14144) 30 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:08.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:08 smithi192 bash[23730]: audit 2023-07-10T16:56:07.622938+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:08.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:08 smithi192 bash[23730]: cephadm 2023-07-10T16:56:07.624042+0000 mgr.y (mgr.14144) 31 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2023-07-10T16:56:08.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:08 smithi192 bash[23730]: audit 2023-07-10T16:56:07.624540+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T16:56:08.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:08 smithi192 bash[23730]: audit 2023-07-10T16:56:07.626031+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T16:56:08.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:08 smithi192 bash[23730]: audit 2023-07-10T16:56:07.627558+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:08.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:08 smithi192 bash[23730]: cephadm 2023-07-10T16:56:07.628892+0000 mgr.y (mgr.14144) 32 : cephadm [INF] Reconfiguring daemon mon.b on smithi192 2023-07-10T16:56:08.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:08 smithi192 bash[23730]: audit 2023-07-10T16:56:08.168299+0000 mon.a (mon.0) 174 : audit [DBG] from='client.? 172.21.15.116:0/3832081129' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:08.904 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-10T16:56:08.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-10T16:56:08.917 INFO:tasks.cephadm:Adding mgr.y on smithi116 2023-07-10T16:56:08.917 INFO:tasks.cephadm:Adding mgr.x on smithi192 2023-07-10T16:56:08.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph orch apply mgr '2;smithi116=y;smithi192=x' 2023-07-10T16:56:10.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:10 smithi116 bash[23740]: cluster 2023-07-10T16:56:09.357189+0000 mgr.y (mgr.14144) 33 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:10.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:10 smithi116 bash[23740]: audit 2023-07-10T16:56:09.731880+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:10.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:10 smithi116 bash[23740]: audit 2023-07-10T16:56:09.733503+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:10.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:10 smithi116 bash[23740]: audit 2023-07-10T16:56:09.743303+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:10.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:10 smithi116 bash[28351]: cluster 2023-07-10T16:56:09.357189+0000 mgr.y (mgr.14144) 33 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:10.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:10 smithi116 bash[28351]: audit 2023-07-10T16:56:09.731880+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:10.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:10 smithi116 bash[28351]: audit 2023-07-10T16:56:09.733503+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:10.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:10 smithi116 bash[28351]: audit 2023-07-10T16:56:09.743303+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:11.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:10 smithi192 bash[23730]: cluster 2023-07-10T16:56:09.357189+0000 mgr.y (mgr.14144) 33 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:11.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:10 smithi192 bash[23730]: audit 2023-07-10T16:56:09.731880+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:11.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:10 smithi192 bash[23730]: audit 2023-07-10T16:56:09.733503+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:11.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:10 smithi192 bash[23730]: audit 2023-07-10T16:56:09.743303+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:12.022 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mgr update... 2023-07-10T16:56:12.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:12 smithi116 bash[23740]: cluster 2023-07-10T16:56:11.357635+0000 mgr.y (mgr.14144) 34 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:12.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:12 smithi116 bash[23740]: audit 2023-07-10T16:56:12.014457+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:12.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:12 smithi116 bash[23740]: audit 2023-07-10T16:56:12.015910+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:12.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:12 smithi116 bash[23740]: audit 2023-07-10T16:56:12.025108+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:12.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:12 smithi116 bash[23740]: audit 2023-07-10T16:56:12.026723+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T16:56:12.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:12 smithi116 bash[23740]: audit 2023-07-10T16:56:12.030711+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-10T16:56:12.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:12 smithi116 bash[23740]: audit 2023-07-10T16:56:12.032078+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T16:56:12.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:12 smithi116 bash[23740]: audit 2023-07-10T16:56:12.033606+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:12.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:12 smithi116 bash[28351]: cluster 2023-07-10T16:56:11.357635+0000 mgr.y (mgr.14144) 34 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:12.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:12 smithi116 bash[28351]: audit 2023-07-10T16:56:12.014457+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:12.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:12 smithi116 bash[28351]: audit 2023-07-10T16:56:12.015910+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:12.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:12 smithi116 bash[28351]: audit 2023-07-10T16:56:12.025108+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:12.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:12 smithi116 bash[28351]: audit 2023-07-10T16:56:12.026723+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T16:56:12.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:12 smithi116 bash[28351]: audit 2023-07-10T16:56:12.030711+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-10T16:56:12.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:12 smithi116 bash[28351]: audit 2023-07-10T16:56:12.032078+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T16:56:12.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:12 smithi116 bash[28351]: audit 2023-07-10T16:56:12.033606+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:12.966 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:12 smithi192 bash[23730]: cluster 2023-07-10T16:56:11.357635+0000 mgr.y (mgr.14144) 34 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:12.967 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:12 smithi192 bash[23730]: audit 2023-07-10T16:56:12.014457+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:12.967 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:12 smithi192 bash[23730]: audit 2023-07-10T16:56:12.015910+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:12.967 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:12 smithi192 bash[23730]: audit 2023-07-10T16:56:12.025108+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:12.967 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:12 smithi192 bash[23730]: audit 2023-07-10T16:56:12.026723+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T16:56:12.967 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:12 smithi192 bash[23730]: audit 2023-07-10T16:56:12.030711+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-10T16:56:12.968 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:12 smithi192 bash[23730]: audit 2023-07-10T16:56:12.032078+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T16:56:12.968 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:12 smithi192 bash[23730]: audit 2023-07-10T16:56:12.033606+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:12.969 DEBUG:teuthology.orchestra.run.smithi192:mgr.x> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.x.service 2023-07-10T16:56:12.972 INFO:tasks.cephadm:Deploying OSDs... 2023-07-10T16:56:12.973 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-07-10T16:56:12.973 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2023-07-10T16:56:12.982 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-10T16:56:12.983 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2023-07-10T16:56:13.036 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-10T16:56:13.037 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-10T16:56:13.037 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 6h/6d Inode: 358 Links: 1 2023-07-10T16:56:13.037 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-10T16:56:13.037 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-07-10 16:51:28.073417074 +0000 2023-07-10T16:56:13.038 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-07-10 16:51:27.797423437 +0000 2023-07-10T16:56:13.039 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-07-10 16:51:27.797423437 +0000 2023-07-10T16:56:13.039 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2023-07-10T16:56:13.039 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-10T16:56:13.052 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-07-10T16:56:13.052 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-07-10T16:56:13.053 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000393475 s, 1.3 MB/s 2023-07-10T16:56:13.054 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-10T16:56:13.106 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2023-07-10T16:56:13.114 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-10T16:56:13.114 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-10T16:56:13.114 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 6h/6d Inode: 366 Links: 1 2023-07-10T16:56:13.114 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-10T16:56:13.114 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-07-10 16:51:28.593405090 +0000 2023-07-10T16:56:13.115 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-07-10 16:51:28.293412003 +0000 2023-07-10T16:56:13.115 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-07-10 16:51:28.293412003 +0000 2023-07-10T16:56:13.115 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2023-07-10T16:56:13.116 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-10T16:56:13.129 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-07-10T16:56:13.130 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-07-10T16:56:13.130 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000406974 s, 1.3 MB/s 2023-07-10T16:56:13.132 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-10T16:56:13.181 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2023-07-10T16:56:13.190 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-10T16:56:13.190 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-10T16:56:13.190 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 6h/6d Inode: 370 Links: 1 2023-07-10T16:56:13.190 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-10T16:56:13.190 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-07-10 16:51:28.833399557 +0000 2023-07-10T16:56:13.190 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-07-10 16:51:28.829399650 +0000 2023-07-10T16:56:13.191 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-07-10 16:51:28.829399650 +0000 2023-07-10T16:56:13.191 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2023-07-10T16:56:13.191 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-10T16:56:13.209 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-07-10T16:56:13.289 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-07-10T16:56:13.289 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000394475 s, 1.3 MB/s 2023-07-10T16:56:13.289 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-10T16:56:13.301 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2023-07-10T16:56:13.321 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-10T16:56:13.321 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-10T16:56:13.322 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 6h/6d Inode: 376 Links: 1 2023-07-10T16:56:13.322 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-10T16:56:13.322 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-07-10 16:51:29.597381947 +0000 2023-07-10T16:56:13.322 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-07-10 16:51:29.317388402 +0000 2023-07-10T16:56:13.322 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-07-10 16:51:29.317388402 +0000 2023-07-10T16:56:13.322 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2023-07-10T16:56:13.323 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-10T16:56:13.379 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-07-10T16:56:13.379 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-07-10T16:56:13.379 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000280133 s, 1.8 MB/s 2023-07-10T16:56:13.380 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-10T16:56:13.389 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-10T16:56:13.390 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2023-07-10T16:56:13.404 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-10T16:56:13.405 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2023-07-10T16:56:13.421 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-10T16:56:13.423 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-10T16:56:13.423 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 361 Links: 1 2023-07-10T16:56:13.424 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-10T16:56:13.424 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-07-10 16:51:29.233864106 +0000 2023-07-10T16:56:13.424 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-07-10 16:51:29.229864198 +0000 2023-07-10T16:56:13.424 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-07-10 16:51:29.229864198 +0000 2023-07-10T16:56:13.424 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-07-10T16:56:13.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-10T16:56:13.483 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-07-10T16:56:13.484 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-07-10T16:56:13.484 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000434433 s, 1.2 MB/s 2023-07-10T16:56:13.486 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-10T16:56:13.501 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2023-07-10T16:56:13.552 INFO:journalctl@ceph.mgr.x.smithi192.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T16:56:13.558 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-10T16:56:13.558 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-10T16:56:13.558 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 364 Links: 1 2023-07-10T16:56:13.559 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-10T16:56:13.559 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-07-10 16:51:30.053845195 +0000 2023-07-10T16:56:13.559 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-07-10 16:51:29.757852023 +0000 2023-07-10T16:56:13.559 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-07-10 16:51:29.757852023 +0000 2023-07-10T16:56:13.559 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-07-10T16:56:13.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-10T16:56:13.572 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-07-10T16:56:13.573 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-07-10T16:56:13.573 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000324295 s, 1.6 MB/s 2023-07-10T16:56:13.574 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-10T16:56:13.633 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2023-07-10T16:56:13.651 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-10T16:56:13.651 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-10T16:56:13.652 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 372 Links: 1 2023-07-10T16:56:13.652 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-10T16:56:13.652 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-07-10 16:51:30.269840215 +0000 2023-07-10T16:56:13.652 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-07-10 16:51:30.265840306 +0000 2023-07-10T16:56:13.652 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-07-10 16:51:30.265840306 +0000 2023-07-10T16:56:13.652 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-07-10T16:56:13.653 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-10T16:56:13.712 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-07-10T16:56:13.713 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-07-10T16:56:13.713 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000429254 s, 1.2 MB/s 2023-07-10T16:56:13.714 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-10T16:56:13.725 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2023-07-10T16:56:13.732 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-10T16:56:13.732 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-10T16:56:13.732 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 376 Links: 1 2023-07-10T16:56:13.733 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-10T16:56:13.733 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-07-10 16:51:31.085821397 +0000 2023-07-10T16:56:13.733 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-07-10 16:51:30.789828223 +0000 2023-07-10T16:56:13.733 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-07-10 16:51:30.789828223 +0000 2023-07-10T16:56:13.733 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-07-10T16:56:13.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-10T16:56:13.784 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:13 smithi192 bash[23730]: audit 2023-07-10T16:56:12.006346+0000 mgr.y (mgr.14144) 35 : audit [DBG] from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi116=y;smithi192=x", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:56:13.784 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:13 smithi192 bash[23730]: cephadm 2023-07-10T16:56:12.007923+0000 mgr.y (mgr.14144) 36 : cephadm [INF] Saving service mgr spec with placement smithi116=y;smithi192=x;count:2 2023-07-10T16:56:13.785 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:13 smithi192 bash[23730]: cephadm 2023-07-10T16:56:12.035086+0000 mgr.y (mgr.14144) 37 : cephadm [INF] Deploying daemon mgr.x on smithi192 2023-07-10T16:56:13.786 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-07-10T16:56:13.787 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-07-10T16:56:13.787 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000385295 s, 1.3 MB/s 2023-07-10T16:56:13.788 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-10T16:56:13.798 INFO:tasks.cephadm:Deploying osd.0 on smithi116 with /dev/vg_nvme/lv_4... 2023-07-10T16:56:13.798 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-10T16:56:13.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:13 smithi116 bash[23740]: audit 2023-07-10T16:56:12.006346+0000 mgr.y (mgr.14144) 35 : audit [DBG] from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi116=y;smithi192=x", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:56:13.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:13 smithi116 bash[23740]: cephadm 2023-07-10T16:56:12.007923+0000 mgr.y (mgr.14144) 36 : cephadm [INF] Saving service mgr spec with placement smithi116=y;smithi192=x;count:2 2023-07-10T16:56:13.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:13 smithi116 bash[23740]: cephadm 2023-07-10T16:56:12.035086+0000 mgr.y (mgr.14144) 37 : cephadm [INF] Deploying daemon mgr.x on smithi192 2023-07-10T16:56:13.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:13 smithi116 bash[28351]: audit 2023-07-10T16:56:12.006346+0000 mgr.y (mgr.14144) 35 : audit [DBG] from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi116=y;smithi192=x", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:56:13.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:13 smithi116 bash[28351]: cephadm 2023-07-10T16:56:12.007923+0000 mgr.y (mgr.14144) 36 : cephadm [INF] Saving service mgr spec with placement smithi116=y;smithi192=x;count:2 2023-07-10T16:56:13.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:13 smithi116 bash[28351]: cephadm 2023-07-10T16:56:12.035086+0000 mgr.y (mgr.14144) 37 : cephadm [INF] Deploying daemon mgr.x on smithi192 2023-07-10T16:56:14.914 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:14 smithi116 bash[23740]: cluster 2023-07-10T16:56:13.358084+0000 mgr.y (mgr.14144) 38 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:14.914 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:14 smithi116 bash[28351]: cluster 2023-07-10T16:56:13.358084+0000 mgr.y (mgr.14144) 38 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:15.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:14 smithi192 bash[23730]: cluster 2023-07-10T16:56:13.358084+0000 mgr.y (mgr.14144) 38 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:15.135 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 16:56:14 smithi192 systemd[1]: Started Ceph mgr.x for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T16:56:15.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:15 smithi192 bash[23730]: audit 2023-07-10T16:56:14.743202+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:15.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:15 smithi192 bash[23730]: audit 2023-07-10T16:56:14.745845+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:15.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:15 smithi192 bash[23730]: cluster 2023-07-10T16:56:15.358598+0000 mgr.y (mgr.14144) 39 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:15.885 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 16:56:15 smithi192 bash[24443]: debug 2023-07-10T16:56:15.709+0000 7f6994567700 1 -- 172.21.15.192:0/4107023261 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5557208304e0 con 0x555720920c00 2023-07-10T16:56:16.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:15 smithi116 bash[23740]: audit 2023-07-10T16:56:14.743202+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:16.149 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:15 smithi116 bash[23740]: audit 2023-07-10T16:56:14.745845+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:16.149 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:15 smithi116 bash[23740]: cluster 2023-07-10T16:56:15.358598+0000 mgr.y (mgr.14144) 39 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:16.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:15 smithi116 bash[28351]: audit 2023-07-10T16:56:14.743202+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:16.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:15 smithi116 bash[28351]: audit 2023-07-10T16:56:14.745845+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:16.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:15 smithi116 bash[28351]: cluster 2023-07-10T16:56:15.358598+0000 mgr.y (mgr.14144) 39 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:17.205 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-10T16:56:17.205 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-10T16:56:17.205 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2023-07-10T16:56:17.206 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2023-07-10T16:56:17.206 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0225464 s, 465 MB/s 2023-07-10T16:56:17.206 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2023-07-10T16:56:17.874 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi116:vg_nvme/lv_4 2023-07-10T16:56:18.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:18 smithi192 bash[23730]: cluster 2023-07-10T16:56:17.359088+0000 mgr.y (mgr.14144) 40 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:18.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:18 smithi192 bash[23730]: audit 2023-07-10T16:56:17.784567+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:18.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:18 smithi192 bash[23730]: audit 2023-07-10T16:56:17.791640+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:18.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:18 smithi192 bash[23730]: audit 2023-07-10T16:56:17.795199+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T16:56:18.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:18 smithi192 bash[23730]: audit 2023-07-10T16:56:17.797040+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T16:56:18.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:18 smithi192 bash[23730]: audit 2023-07-10T16:56:17.798325+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:18.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:18 smithi116 bash[28351]: cluster 2023-07-10T16:56:17.359088+0000 mgr.y (mgr.14144) 40 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:18.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:18 smithi116 bash[28351]: audit 2023-07-10T16:56:17.784567+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:18.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:18 smithi116 bash[28351]: audit 2023-07-10T16:56:17.791640+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:18.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:18 smithi116 bash[28351]: audit 2023-07-10T16:56:17.795199+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T16:56:18.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:18 smithi116 bash[28351]: audit 2023-07-10T16:56:17.797040+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T16:56:18.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:18 smithi116 bash[28351]: audit 2023-07-10T16:56:17.798325+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:18.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:18 smithi116 bash[23740]: cluster 2023-07-10T16:56:17.359088+0000 mgr.y (mgr.14144) 40 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:18.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:18 smithi116 bash[23740]: audit 2023-07-10T16:56:17.784567+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:18.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:18 smithi116 bash[23740]: audit 2023-07-10T16:56:17.791640+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:18.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:18 smithi116 bash[23740]: audit 2023-07-10T16:56:17.795199+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T16:56:18.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:18 smithi116 bash[23740]: audit 2023-07-10T16:56:17.797040+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T16:56:18.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:18 smithi116 bash[23740]: audit 2023-07-10T16:56:17.798325+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:19.879 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:19 smithi116 bash[28351]: cephadm 2023-07-10T16:56:17.794012+0000 mgr.y (mgr.14144) 41 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2023-07-10T16:56:19.880 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:19 smithi116 bash[28351]: cephadm 2023-07-10T16:56:17.799699+0000 mgr.y (mgr.14144) 42 : cephadm [INF] Reconfiguring daemon mgr.y on smithi116 2023-07-10T16:56:19.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:19 smithi116 bash[23740]: cephadm 2023-07-10T16:56:17.794012+0000 mgr.y (mgr.14144) 41 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2023-07-10T16:56:19.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:19 smithi116 bash[23740]: cephadm 2023-07-10T16:56:17.799699+0000 mgr.y (mgr.14144) 42 : cephadm [INF] Reconfiguring daemon mgr.y on smithi116 2023-07-10T16:56:19.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:19 smithi192 bash[23730]: cephadm 2023-07-10T16:56:17.794012+0000 mgr.y (mgr.14144) 41 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2023-07-10T16:56:19.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:19 smithi192 bash[23730]: cephadm 2023-07-10T16:56:17.799699+0000 mgr.y (mgr.14144) 42 : cephadm [INF] Reconfiguring daemon mgr.y on smithi116 2023-07-10T16:56:20.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:20 smithi192 bash[23730]: cluster 2023-07-10T16:56:19.359524+0000 mgr.y (mgr.14144) 43 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:20.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:20 smithi192 bash[23730]: audit 2023-07-10T16:56:19.557509+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T16:56:20.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:20 smithi192 bash[23730]: audit 2023-07-10T16:56:19.559296+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T16:56:20.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:20 smithi116 bash[28351]: cluster 2023-07-10T16:56:19.359524+0000 mgr.y (mgr.14144) 43 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:20.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:20 smithi116 bash[28351]: audit 2023-07-10T16:56:19.557509+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T16:56:20.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:20 smithi116 bash[28351]: audit 2023-07-10T16:56:19.559296+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T16:56:20.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:20 smithi116 bash[23740]: cluster 2023-07-10T16:56:19.359524+0000 mgr.y (mgr.14144) 43 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:20.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:20 smithi116 bash[23740]: audit 2023-07-10T16:56:19.557509+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T16:56:20.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:20 smithi116 bash[23740]: audit 2023-07-10T16:56:19.559296+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T16:56:21.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:21 smithi192 bash[23730]: audit 2023-07-10T16:56:20.531534+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:21.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:21 smithi192 bash[23730]: audit 2023-07-10T16:56:20.532922+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:21.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:21 smithi192 bash[23730]: audit 2023-07-10T16:56:20.539437+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:21.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:21 smithi116 bash[28351]: audit 2023-07-10T16:56:20.531534+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:21.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:21 smithi116 bash[28351]: audit 2023-07-10T16:56:20.532922+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:21.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:21 smithi116 bash[28351]: audit 2023-07-10T16:56:20.539437+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:21.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:21 smithi116 bash[23740]: audit 2023-07-10T16:56:20.531534+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:21.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:21 smithi116 bash[23740]: audit 2023-07-10T16:56:20.532922+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:21.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:21 smithi116 bash[23740]: audit 2023-07-10T16:56:20.539437+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:22.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:22 smithi192 bash[23730]: cluster 2023-07-10T16:56:21.359984+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:22.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:22 smithi192 bash[23730]: cluster 2023-07-10T16:56:21.537007+0000 mon.a (mon.0) 197 : cluster [DBG] Standby manager daemon x started 2023-07-10T16:56:22.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:22 smithi192 bash[23730]: audit 2023-07-10T16:56:21.544300+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.192:0/2316581877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-10T16:56:22.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:22 smithi192 bash[23730]: audit 2023-07-10T16:56:21.547960+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.192:0/2316581877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T16:56:22.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:22 smithi192 bash[23730]: audit 2023-07-10T16:56:21.549786+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.192:0/2316581877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-10T16:56:22.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:22 smithi192 bash[23730]: audit 2023-07-10T16:56:21.550282+0000 mon.b (mon.2) 6 : audit [DBG] from='mgr.? 172.21.15.192:0/2316581877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T16:56:22.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:22 smithi192 bash[23730]: audit 2023-07-10T16:56:21.873086+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:56:22.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:22 smithi192 bash[23730]: audit 2023-07-10T16:56:21.876941+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:56:22.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:22 smithi192 bash[23730]: audit 2023-07-10T16:56:21.878523+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:22.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[23740]: cluster 2023-07-10T16:56:21.359984+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:22.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[23740]: cluster 2023-07-10T16:56:21.537007+0000 mon.a (mon.0) 197 : cluster [DBG] Standby manager daemon x started 2023-07-10T16:56:22.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[23740]: audit 2023-07-10T16:56:21.544300+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.192:0/2316581877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-10T16:56:22.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[23740]: audit 2023-07-10T16:56:21.547960+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.192:0/2316581877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T16:56:22.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[23740]: audit 2023-07-10T16:56:21.549786+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.192:0/2316581877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-10T16:56:22.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[23740]: audit 2023-07-10T16:56:21.550282+0000 mon.b (mon.2) 6 : audit [DBG] from='mgr.? 172.21.15.192:0/2316581877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T16:56:22.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[23740]: audit 2023-07-10T16:56:21.873086+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:56:22.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[23740]: audit 2023-07-10T16:56:21.876941+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:56:22.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[23740]: audit 2023-07-10T16:56:21.878523+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:22.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[28351]: cluster 2023-07-10T16:56:21.359984+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:22.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[28351]: cluster 2023-07-10T16:56:21.537007+0000 mon.a (mon.0) 197 : cluster [DBG] Standby manager daemon x started 2023-07-10T16:56:22.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[28351]: audit 2023-07-10T16:56:21.544300+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.192:0/2316581877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-10T16:56:22.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[28351]: audit 2023-07-10T16:56:21.547960+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.192:0/2316581877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T16:56:22.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[28351]: audit 2023-07-10T16:56:21.549786+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.192:0/2316581877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-10T16:56:22.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[28351]: audit 2023-07-10T16:56:21.550282+0000 mon.b (mon.2) 6 : audit [DBG] from='mgr.? 172.21.15.192:0/2316581877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T16:56:22.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[28351]: audit 2023-07-10T16:56:21.873086+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:56:22.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[28351]: audit 2023-07-10T16:56:21.876941+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:56:22.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:22 smithi116 bash[28351]: audit 2023-07-10T16:56:21.878523+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:23.844 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:23 smithi116 bash[23740]: audit 2023-07-10T16:56:21.871353+0000 mgr.y (mgr.14144) 45 : audit [DBG] from='client.14196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:56:23.845 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:23 smithi116 bash[23740]: cephadm 2023-07-10T16:56:21.876361+0000 mgr.y (mgr.14144) 46 : cephadm [INF] Applying service osd.None on host smithi116... 2023-07-10T16:56:23.845 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:23 smithi116 bash[23740]: cluster 2023-07-10T16:56:22.547825+0000 mon.a (mon.0) 201 : cluster [DBG] mgrmap e14: y(active, since 64s), standbys: x 2023-07-10T16:56:23.845 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:23 smithi116 bash[23740]: audit 2023-07-10T16:56:22.548523+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-10T16:56:23.846 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:23 smithi116 bash[28351]: audit 2023-07-10T16:56:21.871353+0000 mgr.y (mgr.14144) 45 : audit [DBG] from='client.14196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:56:23.846 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:23 smithi116 bash[28351]: cephadm 2023-07-10T16:56:21.876361+0000 mgr.y (mgr.14144) 46 : cephadm [INF] Applying service osd.None on host smithi116... 2023-07-10T16:56:23.846 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:23 smithi116 bash[28351]: cluster 2023-07-10T16:56:22.547825+0000 mon.a (mon.0) 201 : cluster [DBG] mgrmap e14: y(active, since 64s), standbys: x 2023-07-10T16:56:23.846 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:23 smithi116 bash[28351]: audit 2023-07-10T16:56:22.548523+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-10T16:56:23.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:23 smithi192 bash[23730]: audit 2023-07-10T16:56:21.871353+0000 mgr.y (mgr.14144) 45 : audit [DBG] from='client.14196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:56:23.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:23 smithi192 bash[23730]: cephadm 2023-07-10T16:56:21.876361+0000 mgr.y (mgr.14144) 46 : cephadm [INF] Applying service osd.None on host smithi116... 2023-07-10T16:56:23.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:23 smithi192 bash[23730]: cluster 2023-07-10T16:56:22.547825+0000 mon.a (mon.0) 201 : cluster [DBG] mgrmap e14: y(active, since 64s), standbys: x 2023-07-10T16:56:23.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:23 smithi192 bash[23730]: audit 2023-07-10T16:56:22.548523+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-10T16:56:24.836 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:24 smithi116 bash[23740]: cluster 2023-07-10T16:56:23.360433+0000 mgr.y (mgr.14144) 47 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:24.836 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:24 smithi116 bash[28351]: cluster 2023-07-10T16:56:23.360433+0000 mgr.y (mgr.14144) 47 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:24.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:24 smithi192 bash[23730]: cluster 2023-07-10T16:56:23.360433+0000 mgr.y (mgr.14144) 47 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:26.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:26 smithi192 bash[23730]: cluster 2023-07-10T16:56:25.360869+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:26.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:26 smithi192 bash[23730]: audit 2023-07-10T16:56:25.621859+0000 mon.c (mon.1) 4 : audit [INF] from='client.? 172.21.15.116:0/1323410179' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9234f7ee-8413-46af-98b7-6b957be6dd6e"}]: dispatch 2023-07-10T16:56:26.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:26 smithi192 bash[23730]: audit 2023-07-10T16:56:25.622539+0000 mon.a (mon.0) 203 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9234f7ee-8413-46af-98b7-6b957be6dd6e"}]: dispatch 2023-07-10T16:56:26.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:26 smithi192 bash[23730]: audit 2023-07-10T16:56:25.629736+0000 mon.a (mon.0) 204 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9234f7ee-8413-46af-98b7-6b957be6dd6e"}]': finished 2023-07-10T16:56:26.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:26 smithi192 bash[23730]: cluster 2023-07-10T16:56:25.629860+0000 mon.a (mon.0) 205 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2023-07-10T16:56:26.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:26 smithi192 bash[23730]: audit 2023-07-10T16:56:25.629993+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T16:56:26.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:26 smithi192 bash[23730]: audit 2023-07-10T16:56:26.261609+0000 mon.a (mon.0) 207 : audit [DBG] from='client.? 172.21.15.116:0/1069725745' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:56:26.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:26 smithi116 bash[23740]: cluster 2023-07-10T16:56:25.360869+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:26.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:26 smithi116 bash[23740]: audit 2023-07-10T16:56:25.621859+0000 mon.c (mon.1) 4 : audit [INF] from='client.? 172.21.15.116:0/1323410179' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9234f7ee-8413-46af-98b7-6b957be6dd6e"}]: dispatch 2023-07-10T16:56:26.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:26 smithi116 bash[23740]: audit 2023-07-10T16:56:25.622539+0000 mon.a (mon.0) 203 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9234f7ee-8413-46af-98b7-6b957be6dd6e"}]: dispatch 2023-07-10T16:56:26.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:26 smithi116 bash[23740]: audit 2023-07-10T16:56:25.629736+0000 mon.a (mon.0) 204 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9234f7ee-8413-46af-98b7-6b957be6dd6e"}]': finished 2023-07-10T16:56:26.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:26 smithi116 bash[23740]: cluster 2023-07-10T16:56:25.629860+0000 mon.a (mon.0) 205 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2023-07-10T16:56:26.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:26 smithi116 bash[23740]: audit 2023-07-10T16:56:25.629993+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T16:56:26.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:26 smithi116 bash[23740]: audit 2023-07-10T16:56:26.261609+0000 mon.a (mon.0) 207 : audit [DBG] from='client.? 172.21.15.116:0/1069725745' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:56:26.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:26 smithi116 bash[28351]: cluster 2023-07-10T16:56:25.360869+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:26.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:26 smithi116 bash[28351]: audit 2023-07-10T16:56:25.621859+0000 mon.c (mon.1) 4 : audit [INF] from='client.? 172.21.15.116:0/1323410179' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9234f7ee-8413-46af-98b7-6b957be6dd6e"}]: dispatch 2023-07-10T16:56:26.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:26 smithi116 bash[28351]: audit 2023-07-10T16:56:25.622539+0000 mon.a (mon.0) 203 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9234f7ee-8413-46af-98b7-6b957be6dd6e"}]: dispatch 2023-07-10T16:56:26.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:26 smithi116 bash[28351]: audit 2023-07-10T16:56:25.629736+0000 mon.a (mon.0) 204 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9234f7ee-8413-46af-98b7-6b957be6dd6e"}]': finished 2023-07-10T16:56:26.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:26 smithi116 bash[28351]: cluster 2023-07-10T16:56:25.629860+0000 mon.a (mon.0) 205 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2023-07-10T16:56:26.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:26 smithi116 bash[28351]: audit 2023-07-10T16:56:25.629993+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T16:56:26.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:26 smithi116 bash[28351]: audit 2023-07-10T16:56:26.261609+0000 mon.a (mon.0) 207 : audit [DBG] from='client.? 172.21.15.116:0/1069725745' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:56:28.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:28 smithi116 bash[23740]: cluster 2023-07-10T16:56:27.361345+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:28.883 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:28 smithi116 bash[28351]: cluster 2023-07-10T16:56:27.361345+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:28.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:28 smithi192 bash[23730]: cluster 2023-07-10T16:56:27.361345+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:30.714 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:30 smithi116 bash[28351]: cluster 2023-07-10T16:56:29.361800+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:30.715 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:30 smithi116 bash[23740]: cluster 2023-07-10T16:56:29.361800+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:30.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:30 smithi192 bash[23730]: cluster 2023-07-10T16:56:29.361800+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:32.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:32 smithi192 bash[23730]: cluster 2023-07-10T16:56:31.362352+0000 mgr.y (mgr.14144) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:32.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:32 smithi116 bash[23740]: cluster 2023-07-10T16:56:31.362352+0000 mgr.y (mgr.14144) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:32.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:32 smithi116 bash[28351]: cluster 2023-07-10T16:56:31.362352+0000 mgr.y (mgr.14144) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:33.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:33 smithi192 bash[23730]: audit 2023-07-10T16:56:33.237375+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-10T16:56:33.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:33 smithi192 bash[23730]: audit 2023-07-10T16:56:33.239134+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:33.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:33 smithi116 bash[28351]: audit 2023-07-10T16:56:33.237375+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-10T16:56:33.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:33 smithi116 bash[28351]: audit 2023-07-10T16:56:33.239134+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:33.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:33 smithi116 bash[23740]: audit 2023-07-10T16:56:33.237375+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-10T16:56:33.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:33 smithi116 bash[23740]: audit 2023-07-10T16:56:33.239134+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:34.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:34 smithi192 bash[23730]: cephadm 2023-07-10T16:56:33.240735+0000 mgr.y (mgr.14144) 52 : cephadm [INF] Deploying daemon osd.0 on smithi116 2023-07-10T16:56:34.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:34 smithi192 bash[23730]: cluster 2023-07-10T16:56:33.362791+0000 mgr.y (mgr.14144) 53 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:34.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:34 smithi116 bash[23740]: cephadm 2023-07-10T16:56:33.240735+0000 mgr.y (mgr.14144) 52 : cephadm [INF] Deploying daemon osd.0 on smithi116 2023-07-10T16:56:34.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:34 smithi116 bash[23740]: cluster 2023-07-10T16:56:33.362791+0000 mgr.y (mgr.14144) 53 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:34.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:34 smithi116 bash[28351]: cephadm 2023-07-10T16:56:33.240735+0000 mgr.y (mgr.14144) 52 : cephadm [INF] Deploying daemon osd.0 on smithi116 2023-07-10T16:56:34.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:34 smithi116 bash[28351]: cluster 2023-07-10T16:56:33.362791+0000 mgr.y (mgr.14144) 53 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:35.601 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 0 on host 'smithi116' 2023-07-10T16:56:35.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:35 smithi192 bash[23730]: audit 2023-07-10T16:56:35.570567+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:35.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:35 smithi116 bash[23740]: audit 2023-07-10T16:56:35.570567+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:35.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:35 smithi116 bash[28351]: audit 2023-07-10T16:56:35.570567+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:36.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:36 smithi192 bash[23730]: cluster 2023-07-10T16:56:35.363072+0000 mgr.y (mgr.14144) 54 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:36.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:36 smithi192 bash[23730]: audit 2023-07-10T16:56:35.575882+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:36.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:36 smithi192 bash[23730]: audit 2023-07-10T16:56:35.594787+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:36.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:36 smithi116 bash[23740]: cluster 2023-07-10T16:56:35.363072+0000 mgr.y (mgr.14144) 54 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:36.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:36 smithi116 bash[23740]: audit 2023-07-10T16:56:35.575882+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:36.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:36 smithi116 bash[23740]: audit 2023-07-10T16:56:35.594787+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:36.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:36 smithi116 bash[28351]: cluster 2023-07-10T16:56:35.363072+0000 mgr.y (mgr.14144) 54 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:36.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:36 smithi116 bash[28351]: audit 2023-07-10T16:56:35.575882+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:36.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:36 smithi116 bash[28351]: audit 2023-07-10T16:56:35.594787+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:36.918 DEBUG:teuthology.orchestra.run.smithi116:osd.0> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.0.service 2023-07-10T16:56:36.920 INFO:tasks.cephadm:Deploying osd.1 on smithi116 with /dev/vg_nvme/lv_3... 2023-07-10T16:56:36.921 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-10T16:56:38.191 INFO:journalctl@ceph.osd.0.smithi116.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T16:56:38.192 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 16:56:37 smithi116 bash[30789]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-07-10T16:56:38.192 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 16:56:37 smithi116 bash[30789]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2023-07-10T16:56:38.192 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 16:56:37 smithi116 bash[30789]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2023-07-10T16:56:38.192 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 16:56:37 smithi116 bash[30789]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2023-07-10T16:56:38.192 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 16:56:37 smithi116 bash[30789]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-07-10T16:56:38.193 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 16:56:37 smithi116 bash[30789]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-07-10T16:56:38.193 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 16:56:37 smithi116 bash[30789]: --> ceph-volume lvm activate successful for osd ID: 0 2023-07-10T16:56:38.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:38 smithi192 bash[23730]: cluster 2023-07-10T16:56:37.363488+0000 mgr.y (mgr.14144) 55 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:38.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:38 smithi116 bash[23740]: cluster 2023-07-10T16:56:37.363488+0000 mgr.y (mgr.14144) 55 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:38.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:38 smithi116 bash[28351]: cluster 2023-07-10T16:56:37.363488+0000 mgr.y (mgr.14144) 55 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:40.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:40 smithi192 bash[23730]: cluster 2023-07-10T16:56:39.363966+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:40.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:40 smithi192 bash[23730]: audit 2023-07-10T16:56:39.848019+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:40.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:40 smithi116 bash[23740]: cluster 2023-07-10T16:56:39.363966+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:40.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:40 smithi116 bash[23740]: audit 2023-07-10T16:56:39.848019+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:40.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:40 smithi116 bash[28351]: cluster 2023-07-10T16:56:39.363966+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:40.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:40 smithi116 bash[28351]: audit 2023-07-10T16:56:39.848019+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:41.799 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 16:56:41 smithi116 bash[30981]: debug 2023-07-10T16:56:41.488+0000 7f8b9308e700 1 -- 172.21.15.116:0/799773228 <== mon.1 v2:172.21.15.116:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e1aec224e0 con 0x55e1aec9b400 2023-07-10T16:56:42.145 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 16:56:41 smithi116 bash[30981]: debug 2023-07-10T16:56:41.789+0000 7f8b9ab19080 -1 Falling back to public interface 2023-07-10T16:56:42.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:42 smithi116 bash[23740]: cluster 2023-07-10T16:56:41.364459+0000 mgr.y (mgr.14144) 57 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:42.873 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:42 smithi116 bash[28351]: cluster 2023-07-10T16:56:41.364459+0000 mgr.y (mgr.14144) 57 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:42.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:42 smithi192 bash[23730]: cluster 2023-07-10T16:56:41.364459+0000 mgr.y (mgr.14144) 57 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:43.462 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-10T16:56:43.462 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-10T16:56:43.462 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2023-07-10T16:56:43.462 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2023-07-10T16:56:43.463 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.019628 s, 534 MB/s 2023-07-10T16:56:43.463 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2023-07-10T16:56:44.598 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi116:vg_nvme/lv_3 2023-07-10T16:56:44.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:44 smithi192 bash[23730]: cluster 2023-07-10T16:56:43.364892+0000 mgr.y (mgr.14144) 58 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:44.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:44 smithi116 bash[28351]: cluster 2023-07-10T16:56:43.364892+0000 mgr.y (mgr.14144) 58 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:44.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:44 smithi116 bash[23740]: cluster 2023-07-10T16:56:43.364892+0000 mgr.y (mgr.14144) 58 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:45.144 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 16:56:44 smithi116 bash[30981]: debug 2023-07-10T16:56:44.761+0000 7f8b9ab19080 -1 osd.0 0 log_to_monitors {default=true} 2023-07-10T16:56:45.634 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:45 smithi192 bash[23730]: audit 2023-07-10T16:56:44.769374+0000 mon.c (mon.1) 5 : audit [INF] from='osd.0 [v2:172.21.15.116:6802/2297482263,v1:172.21.15.116:6803/2297482263]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-10T16:56:45.634 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:45 smithi192 bash[23730]: audit 2023-07-10T16:56:44.769825+0000 mon.a (mon.0) 214 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-10T16:56:45.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:45 smithi116 bash[23740]: audit 2023-07-10T16:56:44.769374+0000 mon.c (mon.1) 5 : audit [INF] from='osd.0 [v2:172.21.15.116:6802/2297482263,v1:172.21.15.116:6803/2297482263]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-10T16:56:45.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:45 smithi116 bash[23740]: audit 2023-07-10T16:56:44.769825+0000 mon.a (mon.0) 214 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-10T16:56:45.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:45 smithi116 bash[28351]: audit 2023-07-10T16:56:44.769374+0000 mon.c (mon.1) 5 : audit [INF] from='osd.0 [v2:172.21.15.116:6802/2297482263,v1:172.21.15.116:6803/2297482263]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-10T16:56:45.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:45 smithi116 bash[28351]: audit 2023-07-10T16:56:44.769825+0000 mon.a (mon.0) 214 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-10T16:56:46.695 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 16:56:46 smithi116 bash[30981]: debug 2023-07-10T16:56:46.389+0000 7f8b9108a700 -1 osd.0 0 waiting for initial osdmap 2023-07-10T16:56:46.696 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:46 smithi116 bash[28351]: cluster 2023-07-10T16:56:45.365318+0000 mgr.y (mgr.14144) 59 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:46.696 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:46 smithi116 bash[28351]: audit 2023-07-10T16:56:45.382645+0000 mon.a (mon.0) 215 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-10T16:56:46.696 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:46 smithi116 bash[28351]: cluster 2023-07-10T16:56:45.382778+0000 mon.a (mon.0) 216 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2023-07-10T16:56:46.696 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:46 smithi116 bash[28351]: audit 2023-07-10T16:56:45.383049+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T16:56:46.697 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:46 smithi116 bash[28351]: audit 2023-07-10T16:56:45.383651+0000 mon.c (mon.1) 6 : audit [INF] from='osd.0 [v2:172.21.15.116:6802/2297482263,v1:172.21.15.116:6803/2297482263]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:56:46.697 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:46 smithi116 bash[28351]: audit 2023-07-10T16:56:45.384194+0000 mon.a (mon.0) 218 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:56:46.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:46 smithi116 bash[23740]: cluster 2023-07-10T16:56:45.365318+0000 mgr.y (mgr.14144) 59 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:46.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:46 smithi116 bash[23740]: audit 2023-07-10T16:56:45.382645+0000 mon.a (mon.0) 215 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-10T16:56:46.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:46 smithi116 bash[23740]: cluster 2023-07-10T16:56:45.382778+0000 mon.a (mon.0) 216 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2023-07-10T16:56:46.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:46 smithi116 bash[23740]: audit 2023-07-10T16:56:45.383049+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T16:56:46.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:46 smithi116 bash[23740]: audit 2023-07-10T16:56:45.383651+0000 mon.c (mon.1) 6 : audit [INF] from='osd.0 [v2:172.21.15.116:6802/2297482263,v1:172.21.15.116:6803/2297482263]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:56:46.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:46 smithi116 bash[23740]: audit 2023-07-10T16:56:45.384194+0000 mon.a (mon.0) 218 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:56:46.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:46 smithi192 bash[23730]: cluster 2023-07-10T16:56:45.365318+0000 mgr.y (mgr.14144) 59 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:46.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:46 smithi192 bash[23730]: audit 2023-07-10T16:56:45.382645+0000 mon.a (mon.0) 215 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-10T16:56:46.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:46 smithi192 bash[23730]: cluster 2023-07-10T16:56:45.382778+0000 mon.a (mon.0) 216 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2023-07-10T16:56:46.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:46 smithi192 bash[23730]: audit 2023-07-10T16:56:45.383049+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T16:56:46.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:46 smithi192 bash[23730]: audit 2023-07-10T16:56:45.383651+0000 mon.c (mon.1) 6 : audit [INF] from='osd.0 [v2:172.21.15.116:6802/2297482263,v1:172.21.15.116:6803/2297482263]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:56:46.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:46 smithi192 bash[23730]: audit 2023-07-10T16:56:45.384194+0000 mon.a (mon.0) 218 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:56:47.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:47 smithi192 bash[23730]: audit 2023-07-10T16:56:46.386141+0000 mon.a (mon.0) 219 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2023-07-10T16:56:47.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:47 smithi192 bash[23730]: cluster 2023-07-10T16:56:46.386367+0000 mon.a (mon.0) 220 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2023-07-10T16:56:47.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:47 smithi192 bash[23730]: audit 2023-07-10T16:56:46.387187+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T16:56:47.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:47 smithi192 bash[23730]: audit 2023-07-10T16:56:46.394683+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T16:56:47.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:47 smithi116 bash[23740]: audit 2023-07-10T16:56:46.386141+0000 mon.a (mon.0) 219 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2023-07-10T16:56:47.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:47 smithi116 bash[23740]: cluster 2023-07-10T16:56:46.386367+0000 mon.a (mon.0) 220 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2023-07-10T16:56:47.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:47 smithi116 bash[23740]: audit 2023-07-10T16:56:46.387187+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T16:56:47.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:47 smithi116 bash[23740]: audit 2023-07-10T16:56:46.394683+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T16:56:47.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:47 smithi116 bash[28351]: audit 2023-07-10T16:56:46.386141+0000 mon.a (mon.0) 219 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2023-07-10T16:56:47.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:47 smithi116 bash[28351]: cluster 2023-07-10T16:56:46.386367+0000 mon.a (mon.0) 220 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2023-07-10T16:56:47.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:47 smithi116 bash[28351]: audit 2023-07-10T16:56:46.387187+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T16:56:47.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:47 smithi116 bash[28351]: audit 2023-07-10T16:56:46.394683+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T16:56:48.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[28351]: cluster 2023-07-10T16:56:45.784089+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:56:48.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[28351]: cluster 2023-07-10T16:56:45.784373+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:56:48.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[28351]: cluster 2023-07-10T16:56:47.365783+0000 mgr.y (mgr.14144) 60 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:48.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[28351]: cluster 2023-07-10T16:56:47.396727+0000 mon.a (mon.0) 223 : cluster [INF] osd.0 [v2:172.21.15.116:6802/2297482263,v1:172.21.15.116:6803/2297482263] boot 2023-07-10T16:56:48.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[28351]: cluster 2023-07-10T16:56:47.396820+0000 mon.a (mon.0) 224 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2023-07-10T16:56:48.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[28351]: audit 2023-07-10T16:56:47.397614+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T16:56:48.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[28351]: audit 2023-07-10T16:56:47.873146+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:48.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[28351]: audit 2023-07-10T16:56:48.137089+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:48.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[28351]: audit 2023-07-10T16:56:48.139149+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:48.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[28351]: audit 2023-07-10T16:56:48.384364+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:48.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[23740]: cluster 2023-07-10T16:56:45.784089+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:56:48.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[23740]: cluster 2023-07-10T16:56:45.784373+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:56:48.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[23740]: cluster 2023-07-10T16:56:47.365783+0000 mgr.y (mgr.14144) 60 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:48.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[23740]: cluster 2023-07-10T16:56:47.396727+0000 mon.a (mon.0) 223 : cluster [INF] osd.0 [v2:172.21.15.116:6802/2297482263,v1:172.21.15.116:6803/2297482263] boot 2023-07-10T16:56:48.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[23740]: cluster 2023-07-10T16:56:47.396820+0000 mon.a (mon.0) 224 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2023-07-10T16:56:48.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[23740]: audit 2023-07-10T16:56:47.397614+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T16:56:48.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[23740]: audit 2023-07-10T16:56:47.873146+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:48.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[23740]: audit 2023-07-10T16:56:48.137089+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:48.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[23740]: audit 2023-07-10T16:56:48.139149+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:48.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:48 smithi116 bash[23740]: audit 2023-07-10T16:56:48.384364+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:48.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:48 smithi192 bash[23730]: cluster 2023-07-10T16:56:45.784089+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:56:48.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:48 smithi192 bash[23730]: cluster 2023-07-10T16:56:45.784373+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:56:48.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:48 smithi192 bash[23730]: cluster 2023-07-10T16:56:47.365783+0000 mgr.y (mgr.14144) 60 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-10T16:56:48.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:48 smithi192 bash[23730]: cluster 2023-07-10T16:56:47.396727+0000 mon.a (mon.0) 223 : cluster [INF] osd.0 [v2:172.21.15.116:6802/2297482263,v1:172.21.15.116:6803/2297482263] boot 2023-07-10T16:56:48.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:48 smithi192 bash[23730]: cluster 2023-07-10T16:56:47.396820+0000 mon.a (mon.0) 224 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2023-07-10T16:56:48.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:48 smithi192 bash[23730]: audit 2023-07-10T16:56:47.397614+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T16:56:48.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:48 smithi192 bash[23730]: audit 2023-07-10T16:56:47.873146+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:48.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:48 smithi192 bash[23730]: audit 2023-07-10T16:56:48.137089+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:48.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:48 smithi192 bash[23730]: audit 2023-07-10T16:56:48.139149+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:56:48.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:48 smithi192 bash[23730]: audit 2023-07-10T16:56:48.384364+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:56:49.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:49 smithi116 bash[23740]: cephadm 2023-07-10T16:56:47.867034+0000 mgr.y (mgr.14144) 61 : cephadm [INF] Detected new or changed devices on smithi116 2023-07-10T16:56:49.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:49 smithi116 bash[23740]: cluster 2023-07-10T16:56:48.417905+0000 mon.a (mon.0) 230 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2023-07-10T16:56:49.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:49 smithi116 bash[23740]: audit 2023-07-10T16:56:48.557767+0000 mgr.y (mgr.14144) 62 : audit [DBG] from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:56:49.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:49 smithi116 bash[23740]: audit 2023-07-10T16:56:48.558911+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:56:49.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:49 smithi116 bash[23740]: cephadm 2023-07-10T16:56:48.561116+0000 mgr.y (mgr.14144) 63 : cephadm [INF] Applying service osd.None on host smithi116... 2023-07-10T16:56:49.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:49 smithi116 bash[23740]: audit 2023-07-10T16:56:48.561634+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:56:49.729 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:49 smithi116 bash[23740]: audit 2023-07-10T16:56:48.563099+0000 mon.a (mon.0) 233 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:49.729 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:49 smithi116 bash[28351]: cephadm 2023-07-10T16:56:47.867034+0000 mgr.y (mgr.14144) 61 : cephadm [INF] Detected new or changed devices on smithi116 2023-07-10T16:56:49.729 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:49 smithi116 bash[28351]: cluster 2023-07-10T16:56:48.417905+0000 mon.a (mon.0) 230 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2023-07-10T16:56:49.730 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:49 smithi116 bash[28351]: audit 2023-07-10T16:56:48.557767+0000 mgr.y (mgr.14144) 62 : audit [DBG] from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:56:49.730 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:49 smithi116 bash[28351]: audit 2023-07-10T16:56:48.558911+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:56:49.730 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:49 smithi116 bash[28351]: cephadm 2023-07-10T16:56:48.561116+0000 mgr.y (mgr.14144) 63 : cephadm [INF] Applying service osd.None on host smithi116... 2023-07-10T16:56:49.730 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:49 smithi116 bash[28351]: audit 2023-07-10T16:56:48.561634+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:56:49.731 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:49 smithi116 bash[28351]: audit 2023-07-10T16:56:48.563099+0000 mon.a (mon.0) 233 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:49.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:49 smithi192 bash[23730]: cephadm 2023-07-10T16:56:47.867034+0000 mgr.y (mgr.14144) 61 : cephadm [INF] Detected new or changed devices on smithi116 2023-07-10T16:56:49.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:49 smithi192 bash[23730]: cluster 2023-07-10T16:56:48.417905+0000 mon.a (mon.0) 230 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2023-07-10T16:56:49.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:49 smithi192 bash[23730]: audit 2023-07-10T16:56:48.557767+0000 mgr.y (mgr.14144) 62 : audit [DBG] from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:56:49.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:49 smithi192 bash[23730]: audit 2023-07-10T16:56:48.558911+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:56:49.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:49 smithi192 bash[23730]: cephadm 2023-07-10T16:56:48.561116+0000 mgr.y (mgr.14144) 63 : cephadm [INF] Applying service osd.None on host smithi116... 2023-07-10T16:56:49.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:49 smithi192 bash[23730]: audit 2023-07-10T16:56:48.561634+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:56:49.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:49 smithi192 bash[23730]: audit 2023-07-10T16:56:48.563099+0000 mon.a (mon.0) 233 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:56:50.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:50 smithi192 bash[23730]: cluster 2023-07-10T16:56:49.366318+0000 mgr.y (mgr.14144) 64 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:56:50.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:50 smithi116 bash[23740]: cluster 2023-07-10T16:56:49.366318+0000 mgr.y (mgr.14144) 64 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:56:50.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:50 smithi116 bash[28351]: cluster 2023-07-10T16:56:49.366318+0000 mgr.y (mgr.14144) 64 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:56:52.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:52 smithi192 bash[23730]: cluster 2023-07-10T16:56:51.366830+0000 mgr.y (mgr.14144) 65 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:56:52.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:52 smithi192 bash[23730]: audit 2023-07-10T16:56:52.356718+0000 mon.c (mon.1) 7 : audit [INF] from='client.? 172.21.15.116:0/3674865671' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d22b106-7d49-4c50-8c01-c018ae237563"}]: dispatch 2023-07-10T16:56:52.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:52 smithi192 bash[23730]: audit 2023-07-10T16:56:52.357445+0000 mon.a (mon.0) 234 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d22b106-7d49-4c50-8c01-c018ae237563"}]: dispatch 2023-07-10T16:56:52.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:52 smithi192 bash[23730]: audit 2023-07-10T16:56:52.362456+0000 mon.a (mon.0) 235 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d22b106-7d49-4c50-8c01-c018ae237563"}]': finished 2023-07-10T16:56:52.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:52 smithi192 bash[23730]: cluster 2023-07-10T16:56:52.362504+0000 mon.a (mon.0) 236 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2023-07-10T16:56:52.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:52 smithi192 bash[23730]: audit 2023-07-10T16:56:52.362641+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T16:56:52.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:52 smithi116 bash[28351]: cluster 2023-07-10T16:56:51.366830+0000 mgr.y (mgr.14144) 65 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:56:52.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:52 smithi116 bash[28351]: audit 2023-07-10T16:56:52.356718+0000 mon.c (mon.1) 7 : audit [INF] from='client.? 172.21.15.116:0/3674865671' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d22b106-7d49-4c50-8c01-c018ae237563"}]: dispatch 2023-07-10T16:56:52.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:52 smithi116 bash[28351]: audit 2023-07-10T16:56:52.357445+0000 mon.a (mon.0) 234 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d22b106-7d49-4c50-8c01-c018ae237563"}]: dispatch 2023-07-10T16:56:52.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:52 smithi116 bash[28351]: audit 2023-07-10T16:56:52.362456+0000 mon.a (mon.0) 235 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d22b106-7d49-4c50-8c01-c018ae237563"}]': finished 2023-07-10T16:56:52.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:52 smithi116 bash[28351]: cluster 2023-07-10T16:56:52.362504+0000 mon.a (mon.0) 236 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2023-07-10T16:56:52.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:52 smithi116 bash[28351]: audit 2023-07-10T16:56:52.362641+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T16:56:52.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:52 smithi116 bash[23740]: cluster 2023-07-10T16:56:51.366830+0000 mgr.y (mgr.14144) 65 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:56:52.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:52 smithi116 bash[23740]: audit 2023-07-10T16:56:52.356718+0000 mon.c (mon.1) 7 : audit [INF] from='client.? 172.21.15.116:0/3674865671' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d22b106-7d49-4c50-8c01-c018ae237563"}]: dispatch 2023-07-10T16:56:52.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:52 smithi116 bash[23740]: audit 2023-07-10T16:56:52.357445+0000 mon.a (mon.0) 234 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d22b106-7d49-4c50-8c01-c018ae237563"}]: dispatch 2023-07-10T16:56:52.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:52 smithi116 bash[23740]: audit 2023-07-10T16:56:52.362456+0000 mon.a (mon.0) 235 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d22b106-7d49-4c50-8c01-c018ae237563"}]': finished 2023-07-10T16:56:52.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:52 smithi116 bash[23740]: cluster 2023-07-10T16:56:52.362504+0000 mon.a (mon.0) 236 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2023-07-10T16:56:52.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:52 smithi116 bash[23740]: audit 2023-07-10T16:56:52.362641+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T16:56:53.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:53 smithi192 bash[23730]: audit 2023-07-10T16:56:53.021446+0000 mon.a (mon.0) 238 : audit [DBG] from='client.? 172.21.15.116:0/1585468956' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:56:53.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:53 smithi116 bash[23740]: audit 2023-07-10T16:56:53.021446+0000 mon.a (mon.0) 238 : audit [DBG] from='client.? 172.21.15.116:0/1585468956' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:56:53.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:53 smithi116 bash[28351]: audit 2023-07-10T16:56:53.021446+0000 mon.a (mon.0) 238 : audit [DBG] from='client.? 172.21.15.116:0/1585468956' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:56:54.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:54 smithi192 bash[23730]: cluster 2023-07-10T16:56:53.367376+0000 mgr.y (mgr.14144) 66 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:56:54.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:54 smithi116 bash[23740]: cluster 2023-07-10T16:56:53.367376+0000 mgr.y (mgr.14144) 66 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:56:54.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:54 smithi116 bash[28351]: cluster 2023-07-10T16:56:53.367376+0000 mgr.y (mgr.14144) 66 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:56:56.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:56 smithi116 bash[28351]: cluster 2023-07-10T16:56:55.367954+0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:56:56.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:56 smithi116 bash[23740]: cluster 2023-07-10T16:56:55.367954+0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:56:56.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:56 smithi192 bash[23730]: cluster 2023-07-10T16:56:55.367954+0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:56:58.832 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:56:58 smithi116 bash[23740]: cluster 2023-07-10T16:56:57.368523+0000 mgr.y (mgr.14144) 68 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:56:58.833 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:56:58 smithi116 bash[28351]: cluster 2023-07-10T16:56:57.368523+0000 mgr.y (mgr.14144) 68 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:56:58.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:56:58 smithi192 bash[23730]: cluster 2023-07-10T16:56:57.368523+0000 mgr.y (mgr.14144) 68 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:00.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:00 smithi116 bash[23740]: cluster 2023-07-10T16:56:59.369124+0000 mgr.y (mgr.14144) 69 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:00.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:00 smithi116 bash[23740]: audit 2023-07-10T16:57:00.060439+0000 mon.a (mon.0) 239 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-10T16:57:00.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:00 smithi116 bash[23740]: audit 2023-07-10T16:57:00.061934+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:57:00.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:00 smithi116 bash[28351]: cluster 2023-07-10T16:56:59.369124+0000 mgr.y (mgr.14144) 69 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:00.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:00 smithi116 bash[28351]: audit 2023-07-10T16:57:00.060439+0000 mon.a (mon.0) 239 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-10T16:57:00.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:00 smithi116 bash[28351]: audit 2023-07-10T16:57:00.061934+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:57:00.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:00 smithi192 bash[23730]: cluster 2023-07-10T16:56:59.369124+0000 mgr.y (mgr.14144) 69 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:00.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:00 smithi192 bash[23730]: audit 2023-07-10T16:57:00.060439+0000 mon.a (mon.0) 239 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-10T16:57:00.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:00 smithi192 bash[23730]: audit 2023-07-10T16:57:00.061934+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:57:01.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:01 smithi192 bash[23730]: cephadm 2023-07-10T16:57:00.062858+0000 mgr.y (mgr.14144) 70 : cephadm [INF] Deploying daemon osd.1 on smithi116 2023-07-10T16:57:01.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:01 smithi116 bash[23740]: cephadm 2023-07-10T16:57:00.062858+0000 mgr.y (mgr.14144) 70 : cephadm [INF] Deploying daemon osd.1 on smithi116 2023-07-10T16:57:01.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:01 smithi116 bash[28351]: cephadm 2023-07-10T16:57:00.062858+0000 mgr.y (mgr.14144) 70 : cephadm [INF] Deploying daemon osd.1 on smithi116 2023-07-10T16:57:02.462 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 1 on host 'smithi116' 2023-07-10T16:57:02.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:02 smithi192 bash[23730]: cluster 2023-07-10T16:57:01.369674+0000 mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:02.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:02 smithi192 bash[23730]: audit 2023-07-10T16:57:02.444972+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:57:02.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:02 smithi192 bash[23730]: audit 2023-07-10T16:57:02.450150+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:02.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:02 smithi192 bash[23730]: audit 2023-07-10T16:57:02.457503+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:02.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:02 smithi116 bash[23740]: cluster 2023-07-10T16:57:01.369674+0000 mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:02.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:02 smithi116 bash[23740]: audit 2023-07-10T16:57:02.444972+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:57:02.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:02 smithi116 bash[23740]: audit 2023-07-10T16:57:02.450150+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:02.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:02 smithi116 bash[23740]: audit 2023-07-10T16:57:02.457503+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:02.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:02 smithi116 bash[28351]: cluster 2023-07-10T16:57:01.369674+0000 mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:02.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:02 smithi116 bash[28351]: audit 2023-07-10T16:57:02.444972+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:57:02.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:02 smithi116 bash[28351]: audit 2023-07-10T16:57:02.450150+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:02.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:02 smithi116 bash[28351]: audit 2023-07-10T16:57:02.457503+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:03.667 DEBUG:teuthology.orchestra.run.smithi116:osd.1> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.1.service 2023-07-10T16:57:03.712 INFO:tasks.cephadm:Deploying osd.2 on smithi116 with /dev/vg_nvme/lv_2... 2023-07-10T16:57:03.713 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-10T16:57:03.903 INFO:journalctl@ceph.osd.1.smithi116.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T16:57:04.626 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:04 smithi116 bash[28351]: cluster 2023-07-10T16:57:03.370155+0000 mgr.y (mgr.14144) 72 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:04.626 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:04 smithi116 bash[23740]: cluster 2023-07-10T16:57:03.370155+0000 mgr.y (mgr.14144) 72 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:04.627 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 16:57:04 smithi116 bash[33536]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-07-10T16:57:04.627 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 16:57:04 smithi116 bash[33536]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2023-07-10T16:57:04.627 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 16:57:04 smithi116 bash[33536]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2023-07-10T16:57:04.627 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 16:57:04 smithi116 bash[33536]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2023-07-10T16:57:04.627 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 16:57:04 smithi116 bash[33536]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2023-07-10T16:57:04.627 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 16:57:04 smithi116 bash[33536]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-07-10T16:57:04.628 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 16:57:04 smithi116 bash[33536]: --> ceph-volume lvm activate successful for osd ID: 1 2023-07-10T16:57:04.634 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:04 smithi192 bash[23730]: cluster 2023-07-10T16:57:03.370155+0000 mgr.y (mgr.14144) 72 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:06.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:06 smithi116 bash[23740]: cluster 2023-07-10T16:57:05.370759+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:06.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:06 smithi116 bash[28351]: cluster 2023-07-10T16:57:05.370759+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:06.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:06 smithi192 bash[23730]: cluster 2023-07-10T16:57:05.370759+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:07.394 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 16:57:07 smithi116 bash[33698]: debug 2023-07-10T16:57:07.041+0000 7f9dfe4d7700 1 -- 172.21.15.116:0/877124424 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557b8a4be4e0 con 0x557b8a537400 2023-07-10T16:57:07.894 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 16:57:07 smithi116 bash[33698]: debug 2023-07-10T16:57:07.581+0000 7f9e05f62080 -1 Falling back to public interface 2023-07-10T16:57:08.634 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:08 smithi192 bash[23730]: audit 2023-07-10T16:57:07.236386+0000 mon.a (mon.0) 244 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:08.634 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:08 smithi192 bash[23730]: cluster 2023-07-10T16:57:07.371190+0000 mgr.y (mgr.14144) 74 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:08.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:08 smithi116 bash[28351]: audit 2023-07-10T16:57:07.236386+0000 mon.a (mon.0) 244 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:08.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:08 smithi116 bash[28351]: cluster 2023-07-10T16:57:07.371190+0000 mgr.y (mgr.14144) 74 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:08.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:08 smithi116 bash[23740]: audit 2023-07-10T16:57:07.236386+0000 mon.a (mon.0) 244 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:08.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:08 smithi116 bash[23740]: cluster 2023-07-10T16:57:07.371190+0000 mgr.y (mgr.14144) 74 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-10T16:57:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-10T16:57:09.642 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2023-07-10T16:57:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2023-07-10T16:57:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0177552 s, 591 MB/s 2023-07-10T16:57:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2023-07-10T16:57:10.886 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:10 smithi116 bash[28351]: cluster 2023-07-10T16:57:09.371611+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:10.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:10 smithi192 bash[23730]: cluster 2023-07-10T16:57:09.371611+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:10.887 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:10 smithi116 bash[23740]: cluster 2023-07-10T16:57:09.371611+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:10.887 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 16:57:10 smithi116 bash[33698]: debug 2023-07-10T16:57:10.569+0000 7f9e05f62080 -1 osd.1 0 log_to_monitors {default=true} 2023-07-10T16:57:10.888 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi116:vg_nvme/lv_2 2023-07-10T16:57:11.836 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:11 smithi116 bash[23740]: audit 2023-07-10T16:57:10.576995+0000 mon.c (mon.1) 8 : audit [INF] from='osd.1 [v2:172.21.15.116:6810/3104401754,v1:172.21.15.116:6811/3104401754]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-10T16:57:11.836 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:11 smithi116 bash[23740]: audit 2023-07-10T16:57:10.577531+0000 mon.a (mon.0) 245 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-10T16:57:11.837 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:11 smithi116 bash[28351]: audit 2023-07-10T16:57:10.576995+0000 mon.c (mon.1) 8 : audit [INF] from='osd.1 [v2:172.21.15.116:6810/3104401754,v1:172.21.15.116:6811/3104401754]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-10T16:57:11.837 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:11 smithi116 bash[28351]: audit 2023-07-10T16:57:10.577531+0000 mon.a (mon.0) 245 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-10T16:57:11.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:11 smithi192 bash[23730]: audit 2023-07-10T16:57:10.576995+0000 mon.c (mon.1) 8 : audit [INF] from='osd.1 [v2:172.21.15.116:6810/3104401754,v1:172.21.15.116:6811/3104401754]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-10T16:57:11.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:11 smithi192 bash[23730]: audit 2023-07-10T16:57:10.577531+0000 mon.a (mon.0) 245 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-10T16:57:12.582 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:12 smithi116 bash[28351]: cluster 2023-07-10T16:57:11.372085+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:12.583 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:12 smithi116 bash[28351]: audit 2023-07-10T16:57:11.442209+0000 mon.a (mon.0) 246 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-10T16:57:12.583 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:12 smithi116 bash[28351]: cluster 2023-07-10T16:57:11.442355+0000 mon.a (mon.0) 247 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2023-07-10T16:57:12.583 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:12 smithi116 bash[28351]: audit 2023-07-10T16:57:11.442576+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T16:57:12.583 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:12 smithi116 bash[28351]: audit 2023-07-10T16:57:11.443399+0000 mon.c (mon.1) 9 : audit [INF] from='osd.1 [v2:172.21.15.116:6810/3104401754,v1:172.21.15.116:6811/3104401754]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:57:12.583 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:12 smithi116 bash[28351]: audit 2023-07-10T16:57:11.443843+0000 mon.a (mon.0) 249 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:57:12.583 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 16:57:12 smithi116 bash[33698]: debug 2023-07-10T16:57:12.445+0000 7f9dfc4d3700 -1 osd.1 0 waiting for initial osdmap 2023-07-10T16:57:12.584 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:12 smithi116 bash[23740]: cluster 2023-07-10T16:57:11.372085+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:12.584 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:12 smithi116 bash[23740]: audit 2023-07-10T16:57:11.442209+0000 mon.a (mon.0) 246 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-10T16:57:12.584 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:12 smithi116 bash[23740]: cluster 2023-07-10T16:57:11.442355+0000 mon.a (mon.0) 247 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2023-07-10T16:57:12.584 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:12 smithi116 bash[23740]: audit 2023-07-10T16:57:11.442576+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T16:57:12.584 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:12 smithi116 bash[23740]: audit 2023-07-10T16:57:11.443399+0000 mon.c (mon.1) 9 : audit [INF] from='osd.1 [v2:172.21.15.116:6810/3104401754,v1:172.21.15.116:6811/3104401754]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:57:12.585 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:12 smithi116 bash[23740]: audit 2023-07-10T16:57:11.443843+0000 mon.a (mon.0) 249 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:57:12.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:12 smithi192 bash[23730]: cluster 2023-07-10T16:57:11.372085+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:12.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:12 smithi192 bash[23730]: audit 2023-07-10T16:57:11.442209+0000 mon.a (mon.0) 246 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-10T16:57:12.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:12 smithi192 bash[23730]: cluster 2023-07-10T16:57:11.442355+0000 mon.a (mon.0) 247 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2023-07-10T16:57:12.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:12 smithi192 bash[23730]: audit 2023-07-10T16:57:11.442576+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T16:57:12.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:12 smithi192 bash[23730]: audit 2023-07-10T16:57:11.443399+0000 mon.c (mon.1) 9 : audit [INF] from='osd.1 [v2:172.21.15.116:6810/3104401754,v1:172.21.15.116:6811/3104401754]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:57:12.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:12 smithi192 bash[23730]: audit 2023-07-10T16:57:11.443843+0000 mon.a (mon.0) 249 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:57:13.622 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:13 smithi116 bash[28351]: audit 2023-07-10T16:57:12.441784+0000 mon.a (mon.0) 250 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2023-07-10T16:57:13.622 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:13 smithi116 bash[28351]: cluster 2023-07-10T16:57:12.441980+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2023-07-10T16:57:13.622 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:13 smithi116 bash[28351]: audit 2023-07-10T16:57:12.442654+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T16:57:13.623 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:13 smithi116 bash[28351]: audit 2023-07-10T16:57:12.450586+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T16:57:13.623 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:13 smithi116 bash[23740]: audit 2023-07-10T16:57:12.441784+0000 mon.a (mon.0) 250 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2023-07-10T16:57:13.623 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:13 smithi116 bash[23740]: cluster 2023-07-10T16:57:12.441980+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2023-07-10T16:57:13.623 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:13 smithi116 bash[23740]: audit 2023-07-10T16:57:12.442654+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T16:57:13.623 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:13 smithi116 bash[23740]: audit 2023-07-10T16:57:12.450586+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T16:57:13.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:13 smithi192 bash[23730]: audit 2023-07-10T16:57:12.441784+0000 mon.a (mon.0) 250 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2023-07-10T16:57:13.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:13 smithi192 bash[23730]: cluster 2023-07-10T16:57:12.441980+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2023-07-10T16:57:13.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:13 smithi192 bash[23730]: audit 2023-07-10T16:57:12.442654+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T16:57:13.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:13 smithi192 bash[23730]: audit 2023-07-10T16:57:12.450586+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T16:57:14.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:14 smithi192 bash[23730]: cluster 2023-07-10T16:57:11.613500+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:57:14.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:14 smithi192 bash[23730]: cluster 2023-07-10T16:57:11.613835+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:57:14.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:14 smithi192 bash[23730]: cluster 2023-07-10T16:57:13.372522+0000 mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:14.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:14 smithi192 bash[23730]: cluster 2023-07-10T16:57:13.452297+0000 mon.a (mon.0) 254 : cluster [INF] osd.1 [v2:172.21.15.116:6810/3104401754,v1:172.21.15.116:6811/3104401754] boot 2023-07-10T16:57:14.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:14 smithi192 bash[23730]: cluster 2023-07-10T16:57:13.452414+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2023-07-10T16:57:14.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:14 smithi192 bash[23730]: audit 2023-07-10T16:57:13.453319+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T16:57:14.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:14 smithi116 bash[23740]: cluster 2023-07-10T16:57:11.613500+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:57:14.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:14 smithi116 bash[23740]: cluster 2023-07-10T16:57:11.613835+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:57:14.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:14 smithi116 bash[23740]: cluster 2023-07-10T16:57:13.372522+0000 mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:14.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:14 smithi116 bash[23740]: cluster 2023-07-10T16:57:13.452297+0000 mon.a (mon.0) 254 : cluster [INF] osd.1 [v2:172.21.15.116:6810/3104401754,v1:172.21.15.116:6811/3104401754] boot 2023-07-10T16:57:14.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:14 smithi116 bash[23740]: cluster 2023-07-10T16:57:13.452414+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2023-07-10T16:57:14.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:14 smithi116 bash[23740]: audit 2023-07-10T16:57:13.453319+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T16:57:14.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:14 smithi116 bash[28351]: cluster 2023-07-10T16:57:11.613500+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:57:14.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:14 smithi116 bash[28351]: cluster 2023-07-10T16:57:11.613835+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:57:14.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:14 smithi116 bash[28351]: cluster 2023-07-10T16:57:13.372522+0000 mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2023-07-10T16:57:14.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:14 smithi116 bash[28351]: cluster 2023-07-10T16:57:13.452297+0000 mon.a (mon.0) 254 : cluster [INF] osd.1 [v2:172.21.15.116:6810/3104401754,v1:172.21.15.116:6811/3104401754] boot 2023-07-10T16:57:14.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:14 smithi116 bash[28351]: cluster 2023-07-10T16:57:13.452414+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2023-07-10T16:57:14.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:14 smithi116 bash[28351]: audit 2023-07-10T16:57:13.453319+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T16:57:15.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[28351]: cluster 2023-07-10T16:57:14.469068+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2023-07-10T16:57:15.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[28351]: cephadm 2023-07-10T16:57:14.611572+0000 mgr.y (mgr.14144) 78 : cephadm [INF] Detected new or changed devices on smithi116 2023-07-10T16:57:15.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[28351]: audit 2023-07-10T16:57:14.618197+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:15.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[28351]: audit 2023-07-10T16:57:14.624403+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:15.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[28351]: audit 2023-07-10T16:57:14.626195+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:57:15.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[28351]: audit 2023-07-10T16:57:14.631440+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:15.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[28351]: audit 2023-07-10T16:57:15.048819+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:57:15.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[28351]: audit 2023-07-10T16:57:15.052068+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:57:15.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[28351]: audit 2023-07-10T16:57:15.053480+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:57:15.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[23740]: cluster 2023-07-10T16:57:14.469068+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2023-07-10T16:57:15.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[23740]: cephadm 2023-07-10T16:57:14.611572+0000 mgr.y (mgr.14144) 78 : cephadm [INF] Detected new or changed devices on smithi116 2023-07-10T16:57:15.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[23740]: audit 2023-07-10T16:57:14.618197+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:15.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[23740]: audit 2023-07-10T16:57:14.624403+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:15.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[23740]: audit 2023-07-10T16:57:14.626195+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:57:15.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[23740]: audit 2023-07-10T16:57:14.631440+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:15.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[23740]: audit 2023-07-10T16:57:15.048819+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:57:15.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[23740]: audit 2023-07-10T16:57:15.052068+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:57:15.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:15 smithi116 bash[23740]: audit 2023-07-10T16:57:15.053480+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:57:15.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:15 smithi192 bash[23730]: cluster 2023-07-10T16:57:14.469068+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2023-07-10T16:57:15.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:15 smithi192 bash[23730]: cephadm 2023-07-10T16:57:14.611572+0000 mgr.y (mgr.14144) 78 : cephadm [INF] Detected new or changed devices on smithi116 2023-07-10T16:57:15.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:15 smithi192 bash[23730]: audit 2023-07-10T16:57:14.618197+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:15.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:15 smithi192 bash[23730]: audit 2023-07-10T16:57:14.624403+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:15.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:15 smithi192 bash[23730]: audit 2023-07-10T16:57:14.626195+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:57:15.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:15 smithi192 bash[23730]: audit 2023-07-10T16:57:14.631440+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:15.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:15 smithi192 bash[23730]: audit 2023-07-10T16:57:15.048819+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:57:15.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:15 smithi192 bash[23730]: audit 2023-07-10T16:57:15.052068+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:57:15.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:15 smithi192 bash[23730]: audit 2023-07-10T16:57:15.053480+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:57:16.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:16 smithi116 bash[28351]: audit 2023-07-10T16:57:15.047197+0000 mgr.y (mgr.14144) 79 : audit [DBG] from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:57:16.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:16 smithi116 bash[28351]: cephadm 2023-07-10T16:57:15.051522+0000 mgr.y (mgr.14144) 80 : cephadm [INF] Applying service osd.None on host smithi116... 2023-07-10T16:57:16.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:16 smithi116 bash[28351]: cluster 2023-07-10T16:57:15.373021+0000 mgr.y (mgr.14144) 81 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:16.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:16 smithi116 bash[23740]: audit 2023-07-10T16:57:15.047197+0000 mgr.y (mgr.14144) 79 : audit [DBG] from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:57:16.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:16 smithi116 bash[23740]: cephadm 2023-07-10T16:57:15.051522+0000 mgr.y (mgr.14144) 80 : cephadm [INF] Applying service osd.None on host smithi116... 2023-07-10T16:57:16.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:16 smithi116 bash[23740]: cluster 2023-07-10T16:57:15.373021+0000 mgr.y (mgr.14144) 81 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:17.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:16 smithi192 bash[23730]: audit 2023-07-10T16:57:15.047197+0000 mgr.y (mgr.14144) 79 : audit [DBG] from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:57:17.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:16 smithi192 bash[23730]: cephadm 2023-07-10T16:57:15.051522+0000 mgr.y (mgr.14144) 80 : cephadm [INF] Applying service osd.None on host smithi116... 2023-07-10T16:57:17.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:16 smithi192 bash[23730]: cluster 2023-07-10T16:57:15.373021+0000 mgr.y (mgr.14144) 81 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:18.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:18 smithi116 bash[23740]: cluster 2023-07-10T16:57:17.373619+0000 mgr.y (mgr.14144) 82 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:18.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:18 smithi116 bash[28351]: cluster 2023-07-10T16:57:17.373619+0000 mgr.y (mgr.14144) 82 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:19.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:18 smithi192 bash[23730]: cluster 2023-07-10T16:57:17.373619+0000 mgr.y (mgr.14144) 82 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:20.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:19 smithi192 bash[23730]: audit 2023-07-10T16:57:18.895543+0000 mon.c (mon.1) 10 : audit [INF] from='client.? 172.21.15.116:0/560165060' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1368d93-6cb3-4fe8-b945-974ee8982326"}]: dispatch 2023-07-10T16:57:20.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:19 smithi192 bash[23730]: audit 2023-07-10T16:57:18.896092+0000 mon.a (mon.0) 265 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1368d93-6cb3-4fe8-b945-974ee8982326"}]: dispatch 2023-07-10T16:57:20.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:19 smithi192 bash[23730]: audit 2023-07-10T16:57:18.903594+0000 mon.a (mon.0) 266 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1368d93-6cb3-4fe8-b945-974ee8982326"}]': finished 2023-07-10T16:57:20.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:19 smithi192 bash[23730]: cluster 2023-07-10T16:57:18.903722+0000 mon.a (mon.0) 267 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2023-07-10T16:57:20.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:19 smithi192 bash[23730]: audit 2023-07-10T16:57:18.903942+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T16:57:20.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:19 smithi192 bash[23730]: audit 2023-07-10T16:57:19.559664+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T16:57:20.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:19 smithi192 bash[23730]: audit 2023-07-10T16:57:19.561296+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T16:57:20.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:19 smithi192 bash[23730]: audit 2023-07-10T16:57:19.570429+0000 mon.c (mon.1) 11 : audit [DBG] from='client.? 172.21.15.116:0/3656014301' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:57:20.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:19 smithi116 bash[28351]: audit 2023-07-10T16:57:18.895543+0000 mon.c (mon.1) 10 : audit [INF] from='client.? 172.21.15.116:0/560165060' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1368d93-6cb3-4fe8-b945-974ee8982326"}]: dispatch 2023-07-10T16:57:20.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:19 smithi116 bash[28351]: audit 2023-07-10T16:57:18.896092+0000 mon.a (mon.0) 265 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1368d93-6cb3-4fe8-b945-974ee8982326"}]: dispatch 2023-07-10T16:57:20.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:19 smithi116 bash[28351]: audit 2023-07-10T16:57:18.903594+0000 mon.a (mon.0) 266 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1368d93-6cb3-4fe8-b945-974ee8982326"}]': finished 2023-07-10T16:57:20.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:19 smithi116 bash[28351]: cluster 2023-07-10T16:57:18.903722+0000 mon.a (mon.0) 267 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2023-07-10T16:57:20.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:19 smithi116 bash[28351]: audit 2023-07-10T16:57:18.903942+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T16:57:20.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:19 smithi116 bash[28351]: audit 2023-07-10T16:57:19.559664+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T16:57:20.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:19 smithi116 bash[28351]: audit 2023-07-10T16:57:19.561296+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T16:57:20.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:19 smithi116 bash[28351]: audit 2023-07-10T16:57:19.570429+0000 mon.c (mon.1) 11 : audit [DBG] from='client.? 172.21.15.116:0/3656014301' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:57:20.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:19 smithi116 bash[23740]: audit 2023-07-10T16:57:18.895543+0000 mon.c (mon.1) 10 : audit [INF] from='client.? 172.21.15.116:0/560165060' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1368d93-6cb3-4fe8-b945-974ee8982326"}]: dispatch 2023-07-10T16:57:20.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:19 smithi116 bash[23740]: audit 2023-07-10T16:57:18.896092+0000 mon.a (mon.0) 265 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1368d93-6cb3-4fe8-b945-974ee8982326"}]: dispatch 2023-07-10T16:57:20.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:19 smithi116 bash[23740]: audit 2023-07-10T16:57:18.903594+0000 mon.a (mon.0) 266 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1368d93-6cb3-4fe8-b945-974ee8982326"}]': finished 2023-07-10T16:57:20.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:19 smithi116 bash[23740]: cluster 2023-07-10T16:57:18.903722+0000 mon.a (mon.0) 267 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2023-07-10T16:57:20.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:19 smithi116 bash[23740]: audit 2023-07-10T16:57:18.903942+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T16:57:20.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:19 smithi116 bash[23740]: audit 2023-07-10T16:57:19.559664+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T16:57:20.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:19 smithi116 bash[23740]: audit 2023-07-10T16:57:19.561296+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T16:57:20.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:19 smithi116 bash[23740]: audit 2023-07-10T16:57:19.570429+0000 mon.c (mon.1) 11 : audit [DBG] from='client.? 172.21.15.116:0/3656014301' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:57:21.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:20 smithi192 bash[23730]: cluster 2023-07-10T16:57:19.374080+0000 mgr.y (mgr.14144) 83 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:21.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:20 smithi116 bash[23740]: cluster 2023-07-10T16:57:19.374080+0000 mgr.y (mgr.14144) 83 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:21.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:20 smithi116 bash[28351]: cluster 2023-07-10T16:57:19.374080+0000 mgr.y (mgr.14144) 83 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:22.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:22 smithi116 bash[23740]: cluster 2023-07-10T16:57:21.374670+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:22.944 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:22 smithi116 bash[28351]: cluster 2023-07-10T16:57:21.374670+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:23.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:22 smithi192 bash[23730]: cluster 2023-07-10T16:57:21.374670+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:24.634 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:24 smithi192 bash[23730]: cluster 2023-07-10T16:57:23.375161+0000 mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:24.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:24 smithi116 bash[23740]: cluster 2023-07-10T16:57:23.375161+0000 mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:24.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:24 smithi116 bash[28351]: cluster 2023-07-10T16:57:23.375161+0000 mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:26.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:26 smithi116 bash[28351]: cluster 2023-07-10T16:57:25.375670+0000 mgr.y (mgr.14144) 86 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:26.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:26 smithi116 bash[23740]: cluster 2023-07-10T16:57:25.375670+0000 mgr.y (mgr.14144) 86 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:26.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:26 smithi192 bash[23730]: cluster 2023-07-10T16:57:25.375670+0000 mgr.y (mgr.14144) 86 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:27.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:27 smithi116 bash[23740]: audit 2023-07-10T16:57:26.946696+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-10T16:57:27.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:27 smithi116 bash[23740]: audit 2023-07-10T16:57:26.948198+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:57:27.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:27 smithi116 bash[28351]: audit 2023-07-10T16:57:26.946696+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-10T16:57:27.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:27 smithi116 bash[28351]: audit 2023-07-10T16:57:26.948198+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:57:27.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:27 smithi192 bash[23730]: audit 2023-07-10T16:57:26.946696+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-10T16:57:27.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:27 smithi192 bash[23730]: audit 2023-07-10T16:57:26.948198+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:57:28.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:28 smithi192 bash[23730]: cephadm 2023-07-10T16:57:26.949419+0000 mgr.y (mgr.14144) 87 : cephadm [INF] Deploying daemon osd.2 on smithi116 2023-07-10T16:57:28.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:28 smithi192 bash[23730]: cluster 2023-07-10T16:57:27.376173+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:28.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:28 smithi116 bash[23740]: cephadm 2023-07-10T16:57:26.949419+0000 mgr.y (mgr.14144) 87 : cephadm [INF] Deploying daemon osd.2 on smithi116 2023-07-10T16:57:28.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:28 smithi116 bash[23740]: cluster 2023-07-10T16:57:27.376173+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:28.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:28 smithi116 bash[28351]: cephadm 2023-07-10T16:57:26.949419+0000 mgr.y (mgr.14144) 87 : cephadm [INF] Deploying daemon osd.2 on smithi116 2023-07-10T16:57:28.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:28 smithi116 bash[28351]: cluster 2023-07-10T16:57:27.376173+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:29.485 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 2 on host 'smithi116' 2023-07-10T16:57:30.716 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:30 smithi116 bash[28351]: cluster 2023-07-10T16:57:29.376671+0000 mgr.y (mgr.14144) 89 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:30.717 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:30 smithi116 bash[28351]: audit 2023-07-10T16:57:29.469580+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:57:30.717 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:30 smithi116 bash[28351]: audit 2023-07-10T16:57:29.474273+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:30.717 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:30 smithi116 bash[28351]: audit 2023-07-10T16:57:29.480833+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:30.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:30 smithi116 bash[23740]: cluster 2023-07-10T16:57:29.376671+0000 mgr.y (mgr.14144) 89 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:30.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:30 smithi116 bash[23740]: audit 2023-07-10T16:57:29.469580+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:57:30.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:30 smithi116 bash[23740]: audit 2023-07-10T16:57:29.474273+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:30.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:30 smithi116 bash[23740]: audit 2023-07-10T16:57:29.480833+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:30.719 DEBUG:teuthology.orchestra.run.smithi116:osd.2> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.2.service 2023-07-10T16:57:30.722 INFO:tasks.cephadm:Deploying osd.3 on smithi116 with /dev/vg_nvme/lv_1... 2023-07-10T16:57:30.722 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-10T16:57:30.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:30 smithi192 bash[23730]: cluster 2023-07-10T16:57:29.376671+0000 mgr.y (mgr.14144) 89 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:30.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:30 smithi192 bash[23730]: audit 2023-07-10T16:57:29.469580+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:57:30.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:30 smithi192 bash[23730]: audit 2023-07-10T16:57:29.474273+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:30.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:30 smithi192 bash[23730]: audit 2023-07-10T16:57:29.480833+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:31.740 INFO:journalctl@ceph.osd.2.smithi116.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T16:57:32.143 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 16:57:31 smithi116 bash[36276]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-07-10T16:57:32.144 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 16:57:31 smithi116 bash[36276]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2023-07-10T16:57:32.144 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 16:57:31 smithi116 bash[36276]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2023-07-10T16:57:32.144 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 16:57:31 smithi116 bash[36276]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2023-07-10T16:57:32.144 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 16:57:31 smithi116 bash[36276]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2023-07-10T16:57:32.144 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 16:57:31 smithi116 bash[36276]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-07-10T16:57:32.144 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 16:57:31 smithi116 bash[36276]: --> ceph-volume lvm activate successful for osd ID: 2 2023-07-10T16:57:32.738 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:32 smithi116 bash[28351]: cluster 2023-07-10T16:57:31.377171+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:32.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:32 smithi116 bash[23740]: cluster 2023-07-10T16:57:31.377171+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:32.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:32 smithi192 bash[23730]: cluster 2023-07-10T16:57:31.377171+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:34.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:34 smithi116 bash[28351]: cluster 2023-07-10T16:57:33.377597+0000 mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:34.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:34 smithi116 bash[28351]: audit 2023-07-10T16:57:33.710428+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:34.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:34 smithi116 bash[23740]: cluster 2023-07-10T16:57:33.377597+0000 mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:34.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:34 smithi116 bash[23740]: audit 2023-07-10T16:57:33.710428+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:34.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:34 smithi192 bash[23730]: cluster 2023-07-10T16:57:33.377597+0000 mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:34.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:34 smithi192 bash[23730]: audit 2023-07-10T16:57:33.710428+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:35.393 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 16:57:35 smithi116 bash[36504]: debug 2023-07-10T16:57:35.197+0000 7f7e546aa700 1 -- 172.21.15.116:0/53453028 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cc61b044e0 con 0x55cc61b7cc00 2023-07-10T16:57:35.893 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 16:57:35 smithi116 bash[36504]: debug 2023-07-10T16:57:35.765+0000 7f7e5c135080 -1 Falling back to public interface 2023-07-10T16:57:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-10T16:57:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-10T16:57:36.372 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2023-07-10T16:57:36.372 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2023-07-10T16:57:36.372 INFO:teuthology.orchestra.run.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0195839 s, 535 MB/s 2023-07-10T16:57:36.372 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2023-07-10T16:57:36.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:36 smithi116 bash[23740]: cluster 2023-07-10T16:57:35.378099+0000 mgr.y (mgr.14144) 92 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:36.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:36 smithi116 bash[28351]: cluster 2023-07-10T16:57:35.378099+0000 mgr.y (mgr.14144) 92 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:36.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:36 smithi192 bash[23730]: cluster 2023-07-10T16:57:35.378099+0000 mgr.y (mgr.14144) 92 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:37.530 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi116:vg_nvme/lv_1 2023-07-10T16:57:38.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:38 smithi116 bash[23740]: cluster 2023-07-10T16:57:37.378636+0000 mgr.y (mgr.14144) 93 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:38.809 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:38 smithi116 bash[28351]: cluster 2023-07-10T16:57:37.378636+0000 mgr.y (mgr.14144) 93 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:38.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:38 smithi192 bash[23730]: cluster 2023-07-10T16:57:37.378636+0000 mgr.y (mgr.14144) 93 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:39.113 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 16:57:38 smithi116 bash[36504]: debug 2023-07-10T16:57:38.801+0000 7f7e5c135080 -1 osd.2 0 log_to_monitors {default=true} 2023-07-10T16:57:39.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:39 smithi116 bash[23740]: audit 2023-07-10T16:57:38.809792+0000 mon.c (mon.1) 12 : audit [INF] from='osd.2 [v2:172.21.15.116:6818/1907366872,v1:172.21.15.116:6819/1907366872]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-10T16:57:39.686 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:39 smithi116 bash[23740]: audit 2023-07-10T16:57:38.810379+0000 mon.a (mon.0) 277 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-10T16:57:39.686 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:39 smithi116 bash[28351]: audit 2023-07-10T16:57:38.809792+0000 mon.c (mon.1) 12 : audit [INF] from='osd.2 [v2:172.21.15.116:6818/1907366872,v1:172.21.15.116:6819/1907366872]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-10T16:57:39.686 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:39 smithi116 bash[28351]: audit 2023-07-10T16:57:38.810379+0000 mon.a (mon.0) 277 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-10T16:57:39.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:39 smithi192 bash[23730]: audit 2023-07-10T16:57:38.809792+0000 mon.c (mon.1) 12 : audit [INF] from='osd.2 [v2:172.21.15.116:6818/1907366872,v1:172.21.15.116:6819/1907366872]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-10T16:57:39.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:39 smithi192 bash[23730]: audit 2023-07-10T16:57:38.810379+0000 mon.a (mon.0) 277 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-10T16:57:40.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:40 smithi192 bash[23730]: cluster 2023-07-10T16:57:39.379161+0000 mgr.y (mgr.14144) 94 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:40.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:40 smithi192 bash[23730]: audit 2023-07-10T16:57:39.402536+0000 mon.a (mon.0) 278 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-10T16:57:40.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:40 smithi192 bash[23730]: cluster 2023-07-10T16:57:39.402651+0000 mon.a (mon.0) 279 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2023-07-10T16:57:40.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:40 smithi192 bash[23730]: audit 2023-07-10T16:57:39.403025+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T16:57:40.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:40 smithi192 bash[23730]: audit 2023-07-10T16:57:39.405050+0000 mon.c (mon.1) 13 : audit [INF] from='osd.2 [v2:172.21.15.116:6818/1907366872,v1:172.21.15.116:6819/1907366872]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:57:40.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:40 smithi192 bash[23730]: audit 2023-07-10T16:57:39.405648+0000 mon.a (mon.0) 281 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:57:40.893 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 16:57:40 smithi116 bash[36504]: debug 2023-07-10T16:57:40.409+0000 7f7e526a6700 -1 osd.2 0 waiting for initial osdmap 2023-07-10T16:57:40.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:40 smithi116 bash[23740]: cluster 2023-07-10T16:57:39.379161+0000 mgr.y (mgr.14144) 94 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:40.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:40 smithi116 bash[23740]: audit 2023-07-10T16:57:39.402536+0000 mon.a (mon.0) 278 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-10T16:57:40.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:40 smithi116 bash[23740]: cluster 2023-07-10T16:57:39.402651+0000 mon.a (mon.0) 279 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2023-07-10T16:57:40.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:40 smithi116 bash[23740]: audit 2023-07-10T16:57:39.403025+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T16:57:40.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:40 smithi116 bash[23740]: audit 2023-07-10T16:57:39.405050+0000 mon.c (mon.1) 13 : audit [INF] from='osd.2 [v2:172.21.15.116:6818/1907366872,v1:172.21.15.116:6819/1907366872]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:57:40.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:40 smithi116 bash[23740]: audit 2023-07-10T16:57:39.405648+0000 mon.a (mon.0) 281 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:57:40.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:40 smithi116 bash[28351]: cluster 2023-07-10T16:57:39.379161+0000 mgr.y (mgr.14144) 94 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:40.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:40 smithi116 bash[28351]: audit 2023-07-10T16:57:39.402536+0000 mon.a (mon.0) 278 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-10T16:57:40.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:40 smithi116 bash[28351]: cluster 2023-07-10T16:57:39.402651+0000 mon.a (mon.0) 279 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2023-07-10T16:57:40.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:40 smithi116 bash[28351]: audit 2023-07-10T16:57:39.403025+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T16:57:40.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:40 smithi116 bash[28351]: audit 2023-07-10T16:57:39.405050+0000 mon.c (mon.1) 13 : audit [INF] from='osd.2 [v2:172.21.15.116:6818/1907366872,v1:172.21.15.116:6819/1907366872]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:57:40.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:40 smithi116 bash[28351]: audit 2023-07-10T16:57:39.405648+0000 mon.a (mon.0) 281 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:57:41.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:41 smithi192 bash[23730]: audit 2023-07-10T16:57:40.405759+0000 mon.a (mon.0) 282 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2023-07-10T16:57:41.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:41 smithi192 bash[23730]: cluster 2023-07-10T16:57:40.405969+0000 mon.a (mon.0) 283 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2023-07-10T16:57:41.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:41 smithi192 bash[23730]: audit 2023-07-10T16:57:40.406789+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T16:57:41.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:41 smithi192 bash[23730]: audit 2023-07-10T16:57:40.414801+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T16:57:41.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:41 smithi192 bash[23730]: audit 2023-07-10T16:57:40.833161+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:41.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:41 smithi192 bash[23730]: audit 2023-07-10T16:57:40.838540+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:41.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:41 smithi192 bash[23730]: audit 2023-07-10T16:57:40.841168+0000 mon.a (mon.0) 288 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:57:41.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:41 smithi192 bash[23730]: audit 2023-07-10T16:57:40.848277+0000 mon.a (mon.0) 289 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:41.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:41 smithi192 bash[23730]: audit 2023-07-10T16:57:41.319164+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:57:41.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:41 smithi192 bash[23730]: audit 2023-07-10T16:57:41.322222+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:57:41.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:41 smithi192 bash[23730]: audit 2023-07-10T16:57:41.323653+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:57:41.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[23740]: audit 2023-07-10T16:57:40.405759+0000 mon.a (mon.0) 282 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2023-07-10T16:57:41.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[23740]: cluster 2023-07-10T16:57:40.405969+0000 mon.a (mon.0) 283 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2023-07-10T16:57:41.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[23740]: audit 2023-07-10T16:57:40.406789+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T16:57:41.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[23740]: audit 2023-07-10T16:57:40.414801+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T16:57:41.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[23740]: audit 2023-07-10T16:57:40.833161+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:41.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[23740]: audit 2023-07-10T16:57:40.838540+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:41.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[23740]: audit 2023-07-10T16:57:40.841168+0000 mon.a (mon.0) 288 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:57:41.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[23740]: audit 2023-07-10T16:57:40.848277+0000 mon.a (mon.0) 289 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:41.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[23740]: audit 2023-07-10T16:57:41.319164+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:57:41.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[23740]: audit 2023-07-10T16:57:41.322222+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:57:41.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[23740]: audit 2023-07-10T16:57:41.323653+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:57:41.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[28351]: audit 2023-07-10T16:57:40.405759+0000 mon.a (mon.0) 282 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2023-07-10T16:57:41.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[28351]: cluster 2023-07-10T16:57:40.405969+0000 mon.a (mon.0) 283 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2023-07-10T16:57:41.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[28351]: audit 2023-07-10T16:57:40.406789+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T16:57:41.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[28351]: audit 2023-07-10T16:57:40.414801+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T16:57:41.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[28351]: audit 2023-07-10T16:57:40.833161+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:41.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[28351]: audit 2023-07-10T16:57:40.838540+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:41.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[28351]: audit 2023-07-10T16:57:40.841168+0000 mon.a (mon.0) 288 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:57:41.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[28351]: audit 2023-07-10T16:57:40.848277+0000 mon.a (mon.0) 289 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:41.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[28351]: audit 2023-07-10T16:57:41.319164+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:57:41.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[28351]: audit 2023-07-10T16:57:41.322222+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:57:41.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:41 smithi116 bash[28351]: audit 2023-07-10T16:57:41.323653+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:57:42.697 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[28351]: cluster 2023-07-10T16:57:39.815982+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:57:42.697 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[28351]: cluster 2023-07-10T16:57:39.816169+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:57:42.697 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[28351]: cephadm 2023-07-10T16:57:40.824285+0000 mgr.y (mgr.14144) 95 : cephadm [INF] Detected new or changed devices on smithi116 2023-07-10T16:57:42.698 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[28351]: audit 2023-07-10T16:57:41.317576+0000 mgr.y (mgr.14144) 96 : audit [DBG] from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:57:42.698 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[28351]: cephadm 2023-07-10T16:57:41.321742+0000 mgr.y (mgr.14144) 97 : cephadm [INF] Applying service osd.None on host smithi116... 2023-07-10T16:57:42.698 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[28351]: cluster 2023-07-10T16:57:41.379664+0000 mgr.y (mgr.14144) 98 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:42.698 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[28351]: cluster 2023-07-10T16:57:41.415651+0000 mon.a (mon.0) 293 : cluster [INF] osd.2 [v2:172.21.15.116:6818/1907366872,v1:172.21.15.116:6819/1907366872] boot 2023-07-10T16:57:42.698 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[28351]: cluster 2023-07-10T16:57:41.415742+0000 mon.a (mon.0) 294 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2023-07-10T16:57:42.698 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[28351]: audit 2023-07-10T16:57:41.416716+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T16:57:42.699 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[23740]: cluster 2023-07-10T16:57:39.815982+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:57:42.699 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[23740]: cluster 2023-07-10T16:57:39.816169+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:57:42.699 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[23740]: cephadm 2023-07-10T16:57:40.824285+0000 mgr.y (mgr.14144) 95 : cephadm [INF] Detected new or changed devices on smithi116 2023-07-10T16:57:42.699 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[23740]: audit 2023-07-10T16:57:41.317576+0000 mgr.y (mgr.14144) 96 : audit [DBG] from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:57:42.699 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[23740]: cephadm 2023-07-10T16:57:41.321742+0000 mgr.y (mgr.14144) 97 : cephadm [INF] Applying service osd.None on host smithi116... 2023-07-10T16:57:42.700 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[23740]: cluster 2023-07-10T16:57:41.379664+0000 mgr.y (mgr.14144) 98 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:42.700 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[23740]: cluster 2023-07-10T16:57:41.415651+0000 mon.a (mon.0) 293 : cluster [INF] osd.2 [v2:172.21.15.116:6818/1907366872,v1:172.21.15.116:6819/1907366872] boot 2023-07-10T16:57:42.700 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[23740]: cluster 2023-07-10T16:57:41.415742+0000 mon.a (mon.0) 294 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2023-07-10T16:57:42.700 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:42 smithi116 bash[23740]: audit 2023-07-10T16:57:41.416716+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T16:57:42.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:42 smithi192 bash[23730]: cluster 2023-07-10T16:57:39.815982+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:57:42.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:42 smithi192 bash[23730]: cluster 2023-07-10T16:57:39.816169+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:57:42.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:42 smithi192 bash[23730]: cephadm 2023-07-10T16:57:40.824285+0000 mgr.y (mgr.14144) 95 : cephadm [INF] Detected new or changed devices on smithi116 2023-07-10T16:57:42.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:42 smithi192 bash[23730]: audit 2023-07-10T16:57:41.317576+0000 mgr.y (mgr.14144) 96 : audit [DBG] from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:57:42.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:42 smithi192 bash[23730]: cephadm 2023-07-10T16:57:41.321742+0000 mgr.y (mgr.14144) 97 : cephadm [INF] Applying service osd.None on host smithi116... 2023-07-10T16:57:42.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:42 smithi192 bash[23730]: cluster 2023-07-10T16:57:41.379664+0000 mgr.y (mgr.14144) 98 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2023-07-10T16:57:42.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:42 smithi192 bash[23730]: cluster 2023-07-10T16:57:41.415651+0000 mon.a (mon.0) 293 : cluster [INF] osd.2 [v2:172.21.15.116:6818/1907366872,v1:172.21.15.116:6819/1907366872] boot 2023-07-10T16:57:42.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:42 smithi192 bash[23730]: cluster 2023-07-10T16:57:41.415742+0000 mon.a (mon.0) 294 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2023-07-10T16:57:42.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:42 smithi192 bash[23730]: audit 2023-07-10T16:57:41.416716+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T16:57:43.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:43 smithi192 bash[23730]: cluster 2023-07-10T16:57:42.433035+0000 mon.a (mon.0) 296 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2023-07-10T16:57:43.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:43 smithi116 bash[23740]: cluster 2023-07-10T16:57:42.433035+0000 mon.a (mon.0) 296 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2023-07-10T16:57:43.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:43 smithi116 bash[28351]: cluster 2023-07-10T16:57:42.433035+0000 mon.a (mon.0) 296 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2023-07-10T16:57:44.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:44 smithi116 bash[28351]: cluster 2023-07-10T16:57:43.380000+0000 mgr.y (mgr.14144) 99 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:44.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:44 smithi116 bash[23740]: cluster 2023-07-10T16:57:43.380000+0000 mgr.y (mgr.14144) 99 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:44.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:44 smithi192 bash[23730]: cluster 2023-07-10T16:57:43.380000+0000 mgr.y (mgr.14144) 99 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:45.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:45 smithi192 bash[23730]: audit 2023-07-10T16:57:45.201942+0000 mon.c (mon.1) 14 : audit [INF] from='client.? 172.21.15.116:0/3489114927' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d02ea7dd-deab-4a14-b7d2-56e45f1e51e0"}]: dispatch 2023-07-10T16:57:45.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:45 smithi192 bash[23730]: audit 2023-07-10T16:57:45.202727+0000 mon.a (mon.0) 297 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d02ea7dd-deab-4a14-b7d2-56e45f1e51e0"}]: dispatch 2023-07-10T16:57:45.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:45 smithi192 bash[23730]: audit 2023-07-10T16:57:45.212371+0000 mon.a (mon.0) 298 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d02ea7dd-deab-4a14-b7d2-56e45f1e51e0"}]': finished 2023-07-10T16:57:45.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:45 smithi192 bash[23730]: cluster 2023-07-10T16:57:45.212521+0000 mon.a (mon.0) 299 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2023-07-10T16:57:45.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:45 smithi192 bash[23730]: audit 2023-07-10T16:57:45.212777+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:57:45.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:45 smithi116 bash[28351]: audit 2023-07-10T16:57:45.201942+0000 mon.c (mon.1) 14 : audit [INF] from='client.? 172.21.15.116:0/3489114927' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d02ea7dd-deab-4a14-b7d2-56e45f1e51e0"}]: dispatch 2023-07-10T16:57:45.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:45 smithi116 bash[28351]: audit 2023-07-10T16:57:45.202727+0000 mon.a (mon.0) 297 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d02ea7dd-deab-4a14-b7d2-56e45f1e51e0"}]: dispatch 2023-07-10T16:57:45.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:45 smithi116 bash[28351]: audit 2023-07-10T16:57:45.212371+0000 mon.a (mon.0) 298 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d02ea7dd-deab-4a14-b7d2-56e45f1e51e0"}]': finished 2023-07-10T16:57:45.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:45 smithi116 bash[28351]: cluster 2023-07-10T16:57:45.212521+0000 mon.a (mon.0) 299 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2023-07-10T16:57:45.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:45 smithi116 bash[28351]: audit 2023-07-10T16:57:45.212777+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:57:45.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:45 smithi116 bash[23740]: audit 2023-07-10T16:57:45.201942+0000 mon.c (mon.1) 14 : audit [INF] from='client.? 172.21.15.116:0/3489114927' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d02ea7dd-deab-4a14-b7d2-56e45f1e51e0"}]: dispatch 2023-07-10T16:57:45.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:45 smithi116 bash[23740]: audit 2023-07-10T16:57:45.202727+0000 mon.a (mon.0) 297 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d02ea7dd-deab-4a14-b7d2-56e45f1e51e0"}]: dispatch 2023-07-10T16:57:45.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:45 smithi116 bash[23740]: audit 2023-07-10T16:57:45.212371+0000 mon.a (mon.0) 298 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d02ea7dd-deab-4a14-b7d2-56e45f1e51e0"}]': finished 2023-07-10T16:57:45.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:45 smithi116 bash[23740]: cluster 2023-07-10T16:57:45.212521+0000 mon.a (mon.0) 299 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2023-07-10T16:57:45.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:45 smithi116 bash[23740]: audit 2023-07-10T16:57:45.212777+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:57:46.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:46 smithi192 bash[23730]: cluster 2023-07-10T16:57:45.380476+0000 mgr.y (mgr.14144) 100 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:46.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:46 smithi192 bash[23730]: audit 2023-07-10T16:57:45.870143+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.116:0/4083709879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:57:46.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:46 smithi116 bash[28351]: cluster 2023-07-10T16:57:45.380476+0000 mgr.y (mgr.14144) 100 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:46.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:46 smithi116 bash[28351]: audit 2023-07-10T16:57:45.870143+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.116:0/4083709879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:57:46.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:46 smithi116 bash[23740]: cluster 2023-07-10T16:57:45.380476+0000 mgr.y (mgr.14144) 100 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:46.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:46 smithi116 bash[23740]: audit 2023-07-10T16:57:45.870143+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.116:0/4083709879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:57:48.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:48 smithi116 bash[23740]: cluster 2023-07-10T16:57:47.381049+0000 mgr.y (mgr.14144) 101 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:48.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:48 smithi116 bash[28351]: cluster 2023-07-10T16:57:47.381049+0000 mgr.y (mgr.14144) 101 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:48.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:48 smithi192 bash[23730]: cluster 2023-07-10T16:57:47.381049+0000 mgr.y (mgr.14144) 101 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:50.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:50 smithi116 bash[23740]: cluster 2023-07-10T16:57:49.381636+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:50.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:50 smithi116 bash[28351]: cluster 2023-07-10T16:57:49.381636+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:50.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:50 smithi192 bash[23730]: cluster 2023-07-10T16:57:49.381636+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:52.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:52 smithi192 bash[23730]: cluster 2023-07-10T16:57:51.382170+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:52.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:52 smithi116 bash[23740]: cluster 2023-07-10T16:57:51.382170+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:52.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:52 smithi116 bash[28351]: cluster 2023-07-10T16:57:51.382170+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:53.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:53 smithi192 bash[23730]: audit 2023-07-10T16:57:53.291982+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-10T16:57:53.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:53 smithi192 bash[23730]: audit 2023-07-10T16:57:53.293587+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:57:53.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:53 smithi116 bash[28351]: audit 2023-07-10T16:57:53.291982+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-10T16:57:53.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:53 smithi116 bash[28351]: audit 2023-07-10T16:57:53.293587+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:57:53.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:53 smithi116 bash[23740]: audit 2023-07-10T16:57:53.291982+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-10T16:57:53.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:53 smithi116 bash[23740]: audit 2023-07-10T16:57:53.293587+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:57:54.723 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:54 smithi116 bash[28351]: cephadm 2023-07-10T16:57:53.294833+0000 mgr.y (mgr.14144) 104 : cephadm [INF] Deploying daemon osd.3 on smithi116 2023-07-10T16:57:54.724 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:54 smithi116 bash[28351]: cluster 2023-07-10T16:57:53.382684+0000 mgr.y (mgr.14144) 105 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:54.724 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:54 smithi116 bash[23740]: cephadm 2023-07-10T16:57:53.294833+0000 mgr.y (mgr.14144) 104 : cephadm [INF] Deploying daemon osd.3 on smithi116 2023-07-10T16:57:54.724 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:54 smithi116 bash[23740]: cluster 2023-07-10T16:57:53.382684+0000 mgr.y (mgr.14144) 105 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:54.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:54 smithi192 bash[23730]: cephadm 2023-07-10T16:57:53.294833+0000 mgr.y (mgr.14144) 104 : cephadm [INF] Deploying daemon osd.3 on smithi116 2023-07-10T16:57:54.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:54 smithi192 bash[23730]: cluster 2023-07-10T16:57:53.382684+0000 mgr.y (mgr.14144) 105 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:55.925 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 3 on host 'smithi116' 2023-07-10T16:57:56.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:56 smithi192 bash[23730]: cluster 2023-07-10T16:57:55.383252+0000 mgr.y (mgr.14144) 106 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:56.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:56 smithi192 bash[23730]: audit 2023-07-10T16:57:55.900459+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:57:56.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:56 smithi192 bash[23730]: audit 2023-07-10T16:57:55.908172+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:56.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:56 smithi192 bash[23730]: audit 2023-07-10T16:57:55.921289+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:56.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:56 smithi192 bash[23730]: audit 2023-07-10T16:57:55.924139+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2023-07-10T16:57:56.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:56 smithi116 bash[28351]: cluster 2023-07-10T16:57:55.383252+0000 mgr.y (mgr.14144) 106 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:56.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:56 smithi116 bash[28351]: audit 2023-07-10T16:57:55.900459+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:57:56.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:56 smithi116 bash[28351]: audit 2023-07-10T16:57:55.908172+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:56.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:56 smithi116 bash[28351]: audit 2023-07-10T16:57:55.921289+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:56.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:56 smithi116 bash[28351]: audit 2023-07-10T16:57:55.924139+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2023-07-10T16:57:56.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:56 smithi116 bash[23740]: cluster 2023-07-10T16:57:55.383252+0000 mgr.y (mgr.14144) 106 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:56.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:56 smithi116 bash[23740]: audit 2023-07-10T16:57:55.900459+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:57:56.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:56 smithi116 bash[23740]: audit 2023-07-10T16:57:55.908172+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:56.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:56 smithi116 bash[23740]: audit 2023-07-10T16:57:55.921289+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:57:56.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:56 smithi116 bash[23740]: audit 2023-07-10T16:57:55.924139+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2023-07-10T16:57:57.231 DEBUG:teuthology.orchestra.run.smithi116:osd.3> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.3.service 2023-07-10T16:57:57.233 INFO:tasks.cephadm:Deploying osd.4 on smithi192 with /dev/vg_nvme/lv_4... 2023-07-10T16:57:57.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-10T16:57:57.310 INFO:journalctl@ceph.osd.3.smithi116.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T16:57:58.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[23740]: audit 2023-07-10T16:57:56.928187+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2023-07-10T16:57:58.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[23740]: cluster 2023-07-10T16:57:56.928413+0000 mon.a (mon.0) 308 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2023-07-10T16:57:58.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[23740]: audit 2023-07-10T16:57:56.928803+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:57:58.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[23740]: audit 2023-07-10T16:57:56.930786+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2023-07-10T16:57:58.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[23740]: cluster 2023-07-10T16:57:57.383887+0000 mgr.y (mgr.14144) 107 : cluster [DBG] pgmap v90: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:58.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[28351]: audit 2023-07-10T16:57:56.928187+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2023-07-10T16:57:58.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[28351]: cluster 2023-07-10T16:57:56.928413+0000 mon.a (mon.0) 308 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2023-07-10T16:57:58.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[28351]: audit 2023-07-10T16:57:56.928803+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:57:58.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[28351]: audit 2023-07-10T16:57:56.930786+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2023-07-10T16:57:58.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[28351]: cluster 2023-07-10T16:57:57.383887+0000 mgr.y (mgr.14144) 107 : cluster [DBG] pgmap v90: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:58.145 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[39051]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-07-10T16:57:58.145 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[39051]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2023-07-10T16:57:58.145 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[39051]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2023-07-10T16:57:58.145 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[39051]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2023-07-10T16:57:58.145 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[39051]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2023-07-10T16:57:58.145 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[39051]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-07-10T16:57:58.146 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 16:57:57 smithi116 bash[39051]: --> ceph-volume lvm activate successful for osd ID: 3 2023-07-10T16:57:58.254 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:57 smithi192 bash[23730]: audit 2023-07-10T16:57:56.928187+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2023-07-10T16:57:58.254 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:57 smithi192 bash[23730]: cluster 2023-07-10T16:57:56.928413+0000 mon.a (mon.0) 308 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2023-07-10T16:57:58.254 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:57 smithi192 bash[23730]: audit 2023-07-10T16:57:56.928803+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:57:58.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:57 smithi192 bash[23730]: audit 2023-07-10T16:57:56.930786+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2023-07-10T16:57:58.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:57 smithi192 bash[23730]: cluster 2023-07-10T16:57:57.383887+0000 mgr.y (mgr.14144) 107 : cluster [DBG] pgmap v90: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:57:59.385 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:58 smithi192 bash[23730]: audit 2023-07-10T16:57:57.937998+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2023-07-10T16:57:59.385 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:58 smithi192 bash[23730]: cluster 2023-07-10T16:57:57.938264+0000 mon.a (mon.0) 312 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2023-07-10T16:57:59.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:57:58 smithi192 bash[23730]: audit 2023-07-10T16:57:57.938917+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:57:59.392 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:58 smithi116 bash[28351]: audit 2023-07-10T16:57:57.937998+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2023-07-10T16:57:59.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:58 smithi116 bash[28351]: cluster 2023-07-10T16:57:57.938264+0000 mon.a (mon.0) 312 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2023-07-10T16:57:59.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:57:58 smithi116 bash[28351]: audit 2023-07-10T16:57:57.938917+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:57:59.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:58 smithi116 bash[23740]: audit 2023-07-10T16:57:57.937998+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2023-07-10T16:57:59.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:58 smithi116 bash[23740]: cluster 2023-07-10T16:57:57.938264+0000 mon.a (mon.0) 312 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2023-07-10T16:57:59.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:57:58 smithi116 bash[23740]: audit 2023-07-10T16:57:57.938917+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:57:59.755 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 16:57:59 smithi116 bash[39200]: debug 2023-07-10T16:57:59.482+0000 7fbe1c0fa700 1 -- 172.21.15.116:0/1984439219 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ca6abe44e0 con 0x55ca6ac5d400 2023-07-10T16:58:00.142 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 16:57:59 smithi116 bash[39200]: debug 2023-07-10T16:57:59.802+0000 7fbe23b85080 -1 Falling back to public interface 2023-07-10T16:58:00.465 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-10T16:58:00.466 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-10T16:58:00.466 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2023-07-10T16:58:00.466 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2023-07-10T16:58:00.466 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0213351 s, 491 MB/s 2023-07-10T16:58:00.466 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2023-07-10T16:58:00.654 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:00 smithi192 bash[23730]: cluster 2023-07-10T16:57:59.384487+0000 mgr.y (mgr.14144) 108 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:58:00.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:00 smithi192 bash[23730]: cluster 2023-07-10T16:57:59.386756+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2023-07-10T16:58:00.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:00 smithi192 bash[23730]: audit 2023-07-10T16:57:59.386931+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:58:00.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:00 smithi192 bash[23730]: audit 2023-07-10T16:57:59.573429+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:00.839 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:00 smithi116 bash[28351]: cluster 2023-07-10T16:57:59.384487+0000 mgr.y (mgr.14144) 108 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:58:00.840 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:00 smithi116 bash[28351]: cluster 2023-07-10T16:57:59.386756+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2023-07-10T16:58:00.840 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:00 smithi116 bash[28351]: audit 2023-07-10T16:57:59.386931+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:58:00.841 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:00 smithi116 bash[28351]: audit 2023-07-10T16:57:59.573429+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:00.841 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:00 smithi116 bash[23740]: cluster 2023-07-10T16:57:59.384487+0000 mgr.y (mgr.14144) 108 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:58:00.842 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:00 smithi116 bash[23740]: cluster 2023-07-10T16:57:59.386756+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2023-07-10T16:58:00.842 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:00 smithi116 bash[23740]: audit 2023-07-10T16:57:59.386931+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:58:00.842 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:00 smithi116 bash[23740]: audit 2023-07-10T16:57:59.573429+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:01.102 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi192:vg_nvme/lv_4 2023-07-10T16:58:02.766 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:02 smithi116 bash[23740]: cluster 2023-07-10T16:58:01.385107+0000 mgr.y (mgr.14144) 109 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:58:02.767 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:02 smithi116 bash[28351]: cluster 2023-07-10T16:58:01.385107+0000 mgr.y (mgr.14144) 109 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:58:02.793 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:02 smithi192 bash[23730]: cluster 2023-07-10T16:58:01.385107+0000 mgr.y (mgr.14144) 109 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:58:03.142 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 16:58:02 smithi116 bash[39200]: debug 2023-07-10T16:58:02.762+0000 7fbe23b85080 -1 osd.3 0 log_to_monitors {default=true} 2023-07-10T16:58:03.851 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:03 smithi116 bash[28351]: audit 2023-07-10T16:58:02.767298+0000 mon.a (mon.0) 317 : audit [INF] from='osd.3 [v2:172.21.15.116:6826/278136235,v1:172.21.15.116:6827/278136235]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-10T16:58:03.852 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:03 smithi116 bash[23740]: audit 2023-07-10T16:58:02.767298+0000 mon.a (mon.0) 317 : audit [INF] from='osd.3 [v2:172.21.15.116:6826/278136235,v1:172.21.15.116:6827/278136235]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-10T16:58:03.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:03 smithi192 bash[23730]: audit 2023-07-10T16:58:02.767298+0000 mon.a (mon.0) 317 : audit [INF] from='osd.3 [v2:172.21.15.116:6826/278136235,v1:172.21.15.116:6827/278136235]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-10T16:58:04.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:04 smithi192 bash[23730]: cluster 2023-07-10T16:58:03.385744+0000 mgr.y (mgr.14144) 110 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:58:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:04 smithi192 bash[23730]: audit 2023-07-10T16:58:03.412977+0000 mon.a (mon.0) 318 : audit [INF] from='osd.3 [v2:172.21.15.116:6826/278136235,v1:172.21.15.116:6827/278136235]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-10T16:58:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:04 smithi192 bash[23730]: cluster 2023-07-10T16:58:03.413102+0000 mon.a (mon.0) 319 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2023-07-10T16:58:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:04 smithi192 bash[23730]: audit 2023-07-10T16:58:03.413308+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:58:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:04 smithi192 bash[23730]: audit 2023-07-10T16:58:03.413878+0000 mon.a (mon.0) 321 : audit [INF] from='osd.3 [v2:172.21.15.116:6826/278136235,v1:172.21.15.116:6827/278136235]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:58:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:04 smithi192 bash[23730]: audit 2023-07-10T16:58:03.869199+0000 mon.a (mon.0) 322 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:58:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:04 smithi192 bash[23730]: audit 2023-07-10T16:58:03.872916+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:58:04.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:04 smithi192 bash[23730]: audit 2023-07-10T16:58:03.874337+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:58:04.892 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[39200]: debug 2023-07-10T16:58:04.422+0000 7fbe1a0f6700 -1 osd.3 0 waiting for initial osdmap 2023-07-10T16:58:04.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[23740]: cluster 2023-07-10T16:58:03.385744+0000 mgr.y (mgr.14144) 110 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:58:04.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[23740]: audit 2023-07-10T16:58:03.412977+0000 mon.a (mon.0) 318 : audit [INF] from='osd.3 [v2:172.21.15.116:6826/278136235,v1:172.21.15.116:6827/278136235]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-10T16:58:04.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[23740]: cluster 2023-07-10T16:58:03.413102+0000 mon.a (mon.0) 319 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2023-07-10T16:58:04.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[23740]: audit 2023-07-10T16:58:03.413308+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:58:04.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[23740]: audit 2023-07-10T16:58:03.413878+0000 mon.a (mon.0) 321 : audit [INF] from='osd.3 [v2:172.21.15.116:6826/278136235,v1:172.21.15.116:6827/278136235]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:58:04.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[23740]: audit 2023-07-10T16:58:03.869199+0000 mon.a (mon.0) 322 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:58:04.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[23740]: audit 2023-07-10T16:58:03.872916+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:58:04.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[23740]: audit 2023-07-10T16:58:03.874337+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:58:04.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[28351]: cluster 2023-07-10T16:58:03.385744+0000 mgr.y (mgr.14144) 110 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:58:04.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[28351]: audit 2023-07-10T16:58:03.412977+0000 mon.a (mon.0) 318 : audit [INF] from='osd.3 [v2:172.21.15.116:6826/278136235,v1:172.21.15.116:6827/278136235]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-10T16:58:04.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[28351]: cluster 2023-07-10T16:58:03.413102+0000 mon.a (mon.0) 319 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2023-07-10T16:58:04.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[28351]: audit 2023-07-10T16:58:03.413308+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:58:04.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[28351]: audit 2023-07-10T16:58:03.413878+0000 mon.a (mon.0) 321 : audit [INF] from='osd.3 [v2:172.21.15.116:6826/278136235,v1:172.21.15.116:6827/278136235]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T16:58:04.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[28351]: audit 2023-07-10T16:58:03.869199+0000 mon.a (mon.0) 322 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:58:04.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[28351]: audit 2023-07-10T16:58:03.872916+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:58:04.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:04 smithi116 bash[28351]: audit 2023-07-10T16:58:03.874337+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:58:05.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:05 smithi192 bash[23730]: audit 2023-07-10T16:58:03.867684+0000 mgr.y (mgr.14144) 111 : audit [DBG] from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:58:05.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:05 smithi192 bash[23730]: cephadm 2023-07-10T16:58:03.872360+0000 mgr.y (mgr.14144) 112 : cephadm [INF] Applying service osd.None on host smithi192... 2023-07-10T16:58:05.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:05 smithi192 bash[23730]: audit 2023-07-10T16:58:04.415853+0000 mon.a (mon.0) 325 : audit [INF] from='osd.3 [v2:172.21.15.116:6826/278136235,v1:172.21.15.116:6827/278136235]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2023-07-10T16:58:05.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:05 smithi192 bash[23730]: cluster 2023-07-10T16:58:04.416061+0000 mon.a (mon.0) 326 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2023-07-10T16:58:05.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:05 smithi192 bash[23730]: audit 2023-07-10T16:58:04.416963+0000 mon.a (mon.0) 327 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:58:05.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:05 smithi192 bash[23730]: audit 2023-07-10T16:58:04.425698+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:58:05.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:05 smithi192 bash[23730]: audit 2023-07-10T16:58:04.735118+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:05.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:05 smithi192 bash[23730]: audit 2023-07-10T16:58:04.987433+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:05.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:05 smithi192 bash[23730]: audit 2023-07-10T16:58:04.989605+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:58:05.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:05 smithi192 bash[23730]: audit 2023-07-10T16:58:04.997916+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:05.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[23740]: audit 2023-07-10T16:58:03.867684+0000 mgr.y (mgr.14144) 111 : audit [DBG] from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:58:05.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[23740]: cephadm 2023-07-10T16:58:03.872360+0000 mgr.y (mgr.14144) 112 : cephadm [INF] Applying service osd.None on host smithi192... 2023-07-10T16:58:05.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[23740]: audit 2023-07-10T16:58:04.415853+0000 mon.a (mon.0) 325 : audit [INF] from='osd.3 [v2:172.21.15.116:6826/278136235,v1:172.21.15.116:6827/278136235]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2023-07-10T16:58:05.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[23740]: cluster 2023-07-10T16:58:04.416061+0000 mon.a (mon.0) 326 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2023-07-10T16:58:05.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[23740]: audit 2023-07-10T16:58:04.416963+0000 mon.a (mon.0) 327 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:58:05.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[23740]: audit 2023-07-10T16:58:04.425698+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:58:05.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[23740]: audit 2023-07-10T16:58:04.735118+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:05.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[23740]: audit 2023-07-10T16:58:04.987433+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:05.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[23740]: audit 2023-07-10T16:58:04.989605+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:58:05.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[23740]: audit 2023-07-10T16:58:04.997916+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:05.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[28351]: audit 2023-07-10T16:58:03.867684+0000 mgr.y (mgr.14144) 111 : audit [DBG] from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:58:05.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[28351]: cephadm 2023-07-10T16:58:03.872360+0000 mgr.y (mgr.14144) 112 : cephadm [INF] Applying service osd.None on host smithi192... 2023-07-10T16:58:05.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[28351]: audit 2023-07-10T16:58:04.415853+0000 mon.a (mon.0) 325 : audit [INF] from='osd.3 [v2:172.21.15.116:6826/278136235,v1:172.21.15.116:6827/278136235]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2023-07-10T16:58:05.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[28351]: cluster 2023-07-10T16:58:04.416061+0000 mon.a (mon.0) 326 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2023-07-10T16:58:05.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[28351]: audit 2023-07-10T16:58:04.416963+0000 mon.a (mon.0) 327 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:58:05.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[28351]: audit 2023-07-10T16:58:04.425698+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:58:05.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[28351]: audit 2023-07-10T16:58:04.735118+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:05.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[28351]: audit 2023-07-10T16:58:04.987433+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:05.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[28351]: audit 2023-07-10T16:58:04.989605+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:58:05.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:05 smithi116 bash[28351]: audit 2023-07-10T16:58:04.997916+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:06.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:06 smithi192 bash[23730]: cluster 2023-07-10T16:58:03.754260+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:58:06.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:06 smithi192 bash[23730]: cluster 2023-07-10T16:58:03.754467+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:58:06.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:06 smithi192 bash[23730]: cephadm 2023-07-10T16:58:04.724962+0000 mgr.y (mgr.14144) 113 : cephadm [INF] Detected new or changed devices on smithi116 2023-07-10T16:58:06.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:06 smithi192 bash[23730]: cluster 2023-07-10T16:58:05.386362+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:58:06.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:06 smithi192 bash[23730]: cluster 2023-07-10T16:58:05.427711+0000 mon.a (mon.0) 333 : cluster [INF] osd.3 [v2:172.21.15.116:6826/278136235,v1:172.21.15.116:6827/278136235] boot 2023-07-10T16:58:06.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:06 smithi192 bash[23730]: cluster 2023-07-10T16:58:05.427759+0000 mon.a (mon.0) 334 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2023-07-10T16:58:06.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:06 smithi192 bash[23730]: audit 2023-07-10T16:58:05.428557+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:58:06.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:06 smithi116 bash[23740]: cluster 2023-07-10T16:58:03.754260+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:58:06.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:06 smithi116 bash[23740]: cluster 2023-07-10T16:58:03.754467+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:58:06.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:06 smithi116 bash[23740]: cephadm 2023-07-10T16:58:04.724962+0000 mgr.y (mgr.14144) 113 : cephadm [INF] Detected new or changed devices on smithi116 2023-07-10T16:58:06.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:06 smithi116 bash[23740]: cluster 2023-07-10T16:58:05.386362+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:58:06.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:06 smithi116 bash[23740]: cluster 2023-07-10T16:58:05.427711+0000 mon.a (mon.0) 333 : cluster [INF] osd.3 [v2:172.21.15.116:6826/278136235,v1:172.21.15.116:6827/278136235] boot 2023-07-10T16:58:06.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:06 smithi116 bash[23740]: cluster 2023-07-10T16:58:05.427759+0000 mon.a (mon.0) 334 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2023-07-10T16:58:06.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:06 smithi116 bash[23740]: audit 2023-07-10T16:58:05.428557+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:58:06.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:06 smithi116 bash[28351]: cluster 2023-07-10T16:58:03.754260+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:58:06.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:06 smithi116 bash[28351]: cluster 2023-07-10T16:58:03.754467+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:58:06.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:06 smithi116 bash[28351]: cephadm 2023-07-10T16:58:04.724962+0000 mgr.y (mgr.14144) 113 : cephadm [INF] Detected new or changed devices on smithi116 2023-07-10T16:58:06.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:06 smithi116 bash[28351]: cluster 2023-07-10T16:58:05.386362+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-07-10T16:58:06.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:06 smithi116 bash[28351]: cluster 2023-07-10T16:58:05.427711+0000 mon.a (mon.0) 333 : cluster [INF] osd.3 [v2:172.21.15.116:6826/278136235,v1:172.21.15.116:6827/278136235] boot 2023-07-10T16:58:06.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:06 smithi116 bash[28351]: cluster 2023-07-10T16:58:05.427759+0000 mon.a (mon.0) 334 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2023-07-10T16:58:06.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:06 smithi116 bash[28351]: audit 2023-07-10T16:58:05.428557+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T16:58:07.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:07 smithi192 bash[23730]: cluster 2023-07-10T16:58:06.444245+0000 mon.a (mon.0) 336 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2023-07-10T16:58:07.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:07 smithi192 bash[23730]: audit 2023-07-10T16:58:07.289867+0000 mon.a (mon.0) 337 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de3dac78-358c-4e5d-ae57-445a54a5c29a"}]: dispatch 2023-07-10T16:58:07.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:07 smithi192 bash[23730]: audit 2023-07-10T16:58:07.290508+0000 mon.b (mon.2) 7 : audit [INF] from='client.? 172.21.15.192:0/2991615296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de3dac78-358c-4e5d-ae57-445a54a5c29a"}]: dispatch 2023-07-10T16:58:07.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:07 smithi192 bash[23730]: audit 2023-07-10T16:58:07.298711+0000 mon.a (mon.0) 338 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de3dac78-358c-4e5d-ae57-445a54a5c29a"}]': finished 2023-07-10T16:58:07.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:07 smithi192 bash[23730]: cluster 2023-07-10T16:58:07.298854+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2023-07-10T16:58:07.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:07 smithi192 bash[23730]: audit 2023-07-10T16:58:07.299189+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T16:58:07.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:07 smithi116 bash[28351]: cluster 2023-07-10T16:58:06.444245+0000 mon.a (mon.0) 336 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2023-07-10T16:58:07.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:07 smithi116 bash[28351]: audit 2023-07-10T16:58:07.289867+0000 mon.a (mon.0) 337 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de3dac78-358c-4e5d-ae57-445a54a5c29a"}]: dispatch 2023-07-10T16:58:07.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:07 smithi116 bash[28351]: audit 2023-07-10T16:58:07.290508+0000 mon.b (mon.2) 7 : audit [INF] from='client.? 172.21.15.192:0/2991615296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de3dac78-358c-4e5d-ae57-445a54a5c29a"}]: dispatch 2023-07-10T16:58:07.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:07 smithi116 bash[28351]: audit 2023-07-10T16:58:07.298711+0000 mon.a (mon.0) 338 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de3dac78-358c-4e5d-ae57-445a54a5c29a"}]': finished 2023-07-10T16:58:07.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:07 smithi116 bash[28351]: cluster 2023-07-10T16:58:07.298854+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2023-07-10T16:58:07.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:07 smithi116 bash[28351]: audit 2023-07-10T16:58:07.299189+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T16:58:07.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:07 smithi116 bash[23740]: cluster 2023-07-10T16:58:06.444245+0000 mon.a (mon.0) 336 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2023-07-10T16:58:07.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:07 smithi116 bash[23740]: audit 2023-07-10T16:58:07.289867+0000 mon.a (mon.0) 337 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de3dac78-358c-4e5d-ae57-445a54a5c29a"}]: dispatch 2023-07-10T16:58:07.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:07 smithi116 bash[23740]: audit 2023-07-10T16:58:07.290508+0000 mon.b (mon.2) 7 : audit [INF] from='client.? 172.21.15.192:0/2991615296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de3dac78-358c-4e5d-ae57-445a54a5c29a"}]: dispatch 2023-07-10T16:58:07.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:07 smithi116 bash[23740]: audit 2023-07-10T16:58:07.298711+0000 mon.a (mon.0) 338 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de3dac78-358c-4e5d-ae57-445a54a5c29a"}]': finished 2023-07-10T16:58:07.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:07 smithi116 bash[23740]: cluster 2023-07-10T16:58:07.298854+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2023-07-10T16:58:07.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:07 smithi116 bash[23740]: audit 2023-07-10T16:58:07.299189+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T16:58:08.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:08 smithi192 bash[23730]: cluster 2023-07-10T16:58:07.386937+0000 mgr.y (mgr.14144) 115 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:08.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:08 smithi192 bash[23730]: audit 2023-07-10T16:58:07.924216+0000 mon.b (mon.2) 8 : audit [DBG] from='client.? 172.21.15.192:0/111921311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:58:08.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:08 smithi116 bash[23740]: cluster 2023-07-10T16:58:07.386937+0000 mgr.y (mgr.14144) 115 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:08.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:08 smithi116 bash[23740]: audit 2023-07-10T16:58:07.924216+0000 mon.b (mon.2) 8 : audit [DBG] from='client.? 172.21.15.192:0/111921311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:58:08.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:08 smithi116 bash[28351]: cluster 2023-07-10T16:58:07.386937+0000 mgr.y (mgr.14144) 115 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:08.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:08 smithi116 bash[28351]: audit 2023-07-10T16:58:07.924216+0000 mon.b (mon.2) 8 : audit [DBG] from='client.? 172.21.15.192:0/111921311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:58:10.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:10 smithi192 bash[23730]: cluster 2023-07-10T16:58:09.387527+0000 mgr.y (mgr.14144) 116 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:10.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:10 smithi116 bash[23740]: cluster 2023-07-10T16:58:09.387527+0000 mgr.y (mgr.14144) 116 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:10.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:10 smithi116 bash[28351]: cluster 2023-07-10T16:58:09.387527+0000 mgr.y (mgr.14144) 116 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:12.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:12 smithi192 bash[23730]: cluster 2023-07-10T16:58:11.388076+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:12.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:12 smithi116 bash[23740]: cluster 2023-07-10T16:58:11.388076+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:12.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:12 smithi116 bash[28351]: cluster 2023-07-10T16:58:11.388076+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:14.731 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:14 smithi192 bash[23730]: cluster 2023-07-10T16:58:13.388661+0000 mgr.y (mgr.14144) 118 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:14.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:14 smithi116 bash[28351]: cluster 2023-07-10T16:58:13.388661+0000 mgr.y (mgr.14144) 118 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:14.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:14 smithi116 bash[23740]: cluster 2023-07-10T16:58:13.388661+0000 mgr.y (mgr.14144) 118 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:15.733 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:15 smithi192 bash[23730]: audit 2023-07-10T16:58:14.489459+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-10T16:58:15.734 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:15 smithi192 bash[23730]: audit 2023-07-10T16:58:14.491265+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:58:15.734 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:15 smithi192 bash[23730]: cephadm 2023-07-10T16:58:14.492826+0000 mgr.y (mgr.14144) 119 : cephadm [INF] Deploying daemon osd.4 on smithi192 2023-07-10T16:58:15.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:15 smithi116 bash[23740]: audit 2023-07-10T16:58:14.489459+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-10T16:58:15.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:15 smithi116 bash[23740]: audit 2023-07-10T16:58:14.491265+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:58:15.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:15 smithi116 bash[23740]: cephadm 2023-07-10T16:58:14.492826+0000 mgr.y (mgr.14144) 119 : cephadm [INF] Deploying daemon osd.4 on smithi192 2023-07-10T16:58:15.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:15 smithi116 bash[28351]: audit 2023-07-10T16:58:14.489459+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-10T16:58:15.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:15 smithi116 bash[28351]: audit 2023-07-10T16:58:14.491265+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:58:15.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:15 smithi116 bash[28351]: cephadm 2023-07-10T16:58:14.492826+0000 mgr.y (mgr.14144) 119 : cephadm [INF] Deploying daemon osd.4 on smithi192 2023-07-10T16:58:16.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:16 smithi192 bash[23730]: cluster 2023-07-10T16:58:15.389334+0000 mgr.y (mgr.14144) 120 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:16.763 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 4 on host 'smithi192' 2023-07-10T16:58:16.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:16 smithi116 bash[28351]: cluster 2023-07-10T16:58:15.389334+0000 mgr.y (mgr.14144) 120 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:16.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:16 smithi116 bash[23740]: cluster 2023-07-10T16:58:15.389334+0000 mgr.y (mgr.14144) 120 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:17.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:17 smithi192 bash[23730]: audit 2023-07-10T16:58:16.738895+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:58:17.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:17 smithi192 bash[23730]: audit 2023-07-10T16:58:16.749533+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:17.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:17 smithi192 bash[23730]: audit 2023-07-10T16:58:16.758232+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:17.809 DEBUG:teuthology.orchestra.run.smithi192:osd.4> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.4.service 2023-07-10T16:58:17.813 INFO:tasks.cephadm:Deploying osd.5 on smithi192 with /dev/vg_nvme/lv_3... 2023-07-10T16:58:17.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-10T16:58:17.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:17 smithi116 bash[23740]: audit 2023-07-10T16:58:16.738895+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:58:17.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:17 smithi116 bash[23740]: audit 2023-07-10T16:58:16.749533+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:17.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:17 smithi116 bash[23740]: audit 2023-07-10T16:58:16.758232+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:17.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:17 smithi116 bash[28351]: audit 2023-07-10T16:58:16.738895+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:58:17.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:17 smithi116 bash[28351]: audit 2023-07-10T16:58:16.749533+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:17.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:17 smithi116 bash[28351]: audit 2023-07-10T16:58:16.758232+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:18.293 INFO:journalctl@ceph.osd.4.smithi192.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T16:58:18.709 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:18 smithi192 bash[23730]: cluster 2023-07-10T16:58:17.389855+0000 mgr.y (mgr.14144) 121 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:18.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:18 smithi116 bash[23740]: cluster 2023-07-10T16:58:17.389855+0000 mgr.y (mgr.14144) 121 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:18.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:18 smithi116 bash[28351]: cluster 2023-07-10T16:58:17.389855+0000 mgr.y (mgr.14144) 121 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:19.373 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 16:58:19 smithi192 bash[26444]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-07-10T16:58:19.374 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 16:58:19 smithi192 bash[26444]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-4 --no-mon-config 2023-07-10T16:58:19.374 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 16:58:19 smithi192 bash[26444]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4/block 2023-07-10T16:58:19.374 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 16:58:19 smithi192 bash[26444]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block 2023-07-10T16:58:19.374 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 16:58:19 smithi192 bash[26444]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-07-10T16:58:19.374 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 16:58:19 smithi192 bash[26444]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-07-10T16:58:19.374 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 16:58:19 smithi192 bash[26444]: --> ceph-volume lvm activate successful for osd ID: 4 2023-07-10T16:58:20.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:20 smithi192 bash[23730]: cluster 2023-07-10T16:58:19.390469+0000 mgr.y (mgr.14144) 122 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:20.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:20 smithi192 bash[23730]: audit 2023-07-10T16:58:19.561792+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T16:58:20.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:20 smithi192 bash[23730]: audit 2023-07-10T16:58:19.564205+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T16:58:20.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:20 smithi116 bash[23740]: cluster 2023-07-10T16:58:19.390469+0000 mgr.y (mgr.14144) 122 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:20.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:20 smithi116 bash[23740]: audit 2023-07-10T16:58:19.561792+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T16:58:20.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:20 smithi116 bash[23740]: audit 2023-07-10T16:58:19.564205+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T16:58:20.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:20 smithi116 bash[28351]: cluster 2023-07-10T16:58:19.390469+0000 mgr.y (mgr.14144) 122 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:20.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:20 smithi116 bash[28351]: audit 2023-07-10T16:58:19.561792+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T16:58:20.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:20 smithi116 bash[28351]: audit 2023-07-10T16:58:19.564205+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T16:58:21.635 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 16:58:21 smithi192 bash[26676]: debug 2023-07-10T16:58:21.369+0000 7fbb602af700 1 -- 172.21.15.192:0/1202070767 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55faf17024e0 con 0x55faf177ac00 2023-07-10T16:58:21.986 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-10T16:58:21.986 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-10T16:58:21.986 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2023-07-10T16:58:21.986 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2023-07-10T16:58:21.986 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144606 s, 725 MB/s 2023-07-10T16:58:21.987 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2023-07-10T16:58:22.385 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 16:58:21 smithi192 bash[26676]: debug 2023-07-10T16:58:21.921+0000 7fbb67d3a080 -1 Falling back to public interface 2023-07-10T16:58:22.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:22 smithi116 bash[28351]: cluster 2023-07-10T16:58:21.391050+0000 mgr.y (mgr.14144) 123 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:22.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:22 smithi116 bash[28351]: audit 2023-07-10T16:58:21.463842+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:22.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:22 smithi116 bash[23740]: cluster 2023-07-10T16:58:21.391050+0000 mgr.y (mgr.14144) 123 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:22.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:22 smithi116 bash[23740]: audit 2023-07-10T16:58:21.463842+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:22.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:22 smithi192 bash[23730]: cluster 2023-07-10T16:58:21.391050+0000 mgr.y (mgr.14144) 123 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:22.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:22 smithi192 bash[23730]: audit 2023-07-10T16:58:21.463842+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:23.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi192:vg_nvme/lv_3 2023-07-10T16:58:24.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:24 smithi192 bash[23730]: cluster 2023-07-10T16:58:23.391656+0000 mgr.y (mgr.14144) 124 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:24.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:24 smithi116 bash[23740]: cluster 2023-07-10T16:58:23.391656+0000 mgr.y (mgr.14144) 124 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:24.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:24 smithi116 bash[28351]: cluster 2023-07-10T16:58:23.391656+0000 mgr.y (mgr.14144) 124 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:25.135 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 16:58:24 smithi192 bash[26676]: debug 2023-07-10T16:58:24.921+0000 7fbb67d3a080 -1 osd.4 0 log_to_monitors {default=true} 2023-07-10T16:58:25.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:25 smithi192 bash[23730]: audit 2023-07-10T16:58:24.933191+0000 mon.a (mon.0) 349 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-10T16:58:25.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:25 smithi192 bash[23730]: audit 2023-07-10T16:58:24.933409+0000 mon.b (mon.2) 9 : audit [INF] from='osd.4 [v2:172.21.15.192:6800/1383435807,v1:172.21.15.192:6801/1383435807]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-10T16:58:25.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:25 smithi116 bash[23740]: audit 2023-07-10T16:58:24.933191+0000 mon.a (mon.0) 349 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-10T16:58:25.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:25 smithi116 bash[23740]: audit 2023-07-10T16:58:24.933409+0000 mon.b (mon.2) 9 : audit [INF] from='osd.4 [v2:172.21.15.192:6800/1383435807,v1:172.21.15.192:6801/1383435807]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-10T16:58:25.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:25 smithi116 bash[28351]: audit 2023-07-10T16:58:24.933191+0000 mon.a (mon.0) 349 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-10T16:58:25.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:25 smithi116 bash[28351]: audit 2023-07-10T16:58:24.933409+0000 mon.b (mon.2) 9 : audit [INF] from='osd.4 [v2:172.21.15.192:6800/1383435807,v1:172.21.15.192:6801/1383435807]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-10T16:58:26.885 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 16:58:26 smithi192 bash[26676]: debug 2023-07-10T16:58:26.417+0000 7fbb5e2ab700 -1 osd.4 0 waiting for initial osdmap 2023-07-10T16:58:26.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:26 smithi192 bash[23730]: cluster 2023-07-10T16:58:25.392268+0000 mgr.y (mgr.14144) 125 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:26.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:26 smithi192 bash[23730]: audit 2023-07-10T16:58:25.408968+0000 mon.a (mon.0) 350 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-10T16:58:26.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:26 smithi192 bash[23730]: cluster 2023-07-10T16:58:25.409053+0000 mon.a (mon.0) 351 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2023-07-10T16:58:26.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:26 smithi192 bash[23730]: audit 2023-07-10T16:58:25.409237+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T16:58:26.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:26 smithi192 bash[23730]: audit 2023-07-10T16:58:25.412211+0000 mon.a (mon.0) 353 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:58:26.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:26 smithi192 bash[23730]: audit 2023-07-10T16:58:25.412510+0000 mon.b (mon.2) 10 : audit [INF] from='osd.4 [v2:172.21.15.192:6800/1383435807,v1:172.21.15.192:6801/1383435807]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:58:26.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:26 smithi116 bash[23740]: cluster 2023-07-10T16:58:25.392268+0000 mgr.y (mgr.14144) 125 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:26.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:26 smithi116 bash[23740]: audit 2023-07-10T16:58:25.408968+0000 mon.a (mon.0) 350 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-10T16:58:26.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:26 smithi116 bash[23740]: cluster 2023-07-10T16:58:25.409053+0000 mon.a (mon.0) 351 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2023-07-10T16:58:26.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:26 smithi116 bash[23740]: audit 2023-07-10T16:58:25.409237+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T16:58:26.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:26 smithi116 bash[23740]: audit 2023-07-10T16:58:25.412211+0000 mon.a (mon.0) 353 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:58:26.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:26 smithi116 bash[23740]: audit 2023-07-10T16:58:25.412510+0000 mon.b (mon.2) 10 : audit [INF] from='osd.4 [v2:172.21.15.192:6800/1383435807,v1:172.21.15.192:6801/1383435807]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:58:26.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:26 smithi116 bash[28351]: cluster 2023-07-10T16:58:25.392268+0000 mgr.y (mgr.14144) 125 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:26.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:26 smithi116 bash[28351]: audit 2023-07-10T16:58:25.408968+0000 mon.a (mon.0) 350 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-10T16:58:26.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:26 smithi116 bash[28351]: cluster 2023-07-10T16:58:25.409053+0000 mon.a (mon.0) 351 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2023-07-10T16:58:26.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:26 smithi116 bash[28351]: audit 2023-07-10T16:58:25.409237+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T16:58:26.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:26 smithi116 bash[28351]: audit 2023-07-10T16:58:25.412211+0000 mon.a (mon.0) 353 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:58:26.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:26 smithi116 bash[28351]: audit 2023-07-10T16:58:25.412510+0000 mon.b (mon.2) 10 : audit [INF] from='osd.4 [v2:172.21.15.192:6800/1383435807,v1:172.21.15.192:6801/1383435807]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:58:27.862 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:27 smithi192 bash[23730]: audit 2023-07-10T16:58:26.414142+0000 mon.a (mon.0) 354 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-10T16:58:27.862 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:27 smithi192 bash[23730]: cluster 2023-07-10T16:58:26.414406+0000 mon.a (mon.0) 355 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2023-07-10T16:58:27.862 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:27 smithi192 bash[23730]: audit 2023-07-10T16:58:26.415568+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T16:58:27.862 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:27 smithi192 bash[23730]: audit 2023-07-10T16:58:26.422039+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T16:58:27.863 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:27 smithi192 bash[23730]: audit 2023-07-10T16:58:27.323081+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:58:27.863 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:27 smithi192 bash[23730]: audit 2023-07-10T16:58:27.326786+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:58:27.863 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:27 smithi192 bash[23730]: audit 2023-07-10T16:58:27.328809+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:58:27.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:27 smithi116 bash[23740]: audit 2023-07-10T16:58:26.414142+0000 mon.a (mon.0) 354 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-10T16:58:27.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:27 smithi116 bash[23740]: cluster 2023-07-10T16:58:26.414406+0000 mon.a (mon.0) 355 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2023-07-10T16:58:27.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:27 smithi116 bash[23740]: audit 2023-07-10T16:58:26.415568+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T16:58:27.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:27 smithi116 bash[23740]: audit 2023-07-10T16:58:26.422039+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T16:58:27.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:27 smithi116 bash[23740]: audit 2023-07-10T16:58:27.323081+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:58:27.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:27 smithi116 bash[23740]: audit 2023-07-10T16:58:27.326786+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:58:27.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:27 smithi116 bash[23740]: audit 2023-07-10T16:58:27.328809+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:58:27.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:27 smithi116 bash[28351]: audit 2023-07-10T16:58:26.414142+0000 mon.a (mon.0) 354 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-10T16:58:27.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:27 smithi116 bash[28351]: cluster 2023-07-10T16:58:26.414406+0000 mon.a (mon.0) 355 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2023-07-10T16:58:27.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:27 smithi116 bash[28351]: audit 2023-07-10T16:58:26.415568+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T16:58:27.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:27 smithi116 bash[28351]: audit 2023-07-10T16:58:26.422039+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T16:58:27.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:27 smithi116 bash[28351]: audit 2023-07-10T16:58:27.323081+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:58:27.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:27 smithi116 bash[28351]: audit 2023-07-10T16:58:27.326786+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:58:27.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:27 smithi116 bash[28351]: audit 2023-07-10T16:58:27.328809+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:58:28.732 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:28 smithi192 bash[23730]: cluster 2023-07-10T16:58:25.910645+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:58:28.732 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:28 smithi192 bash[23730]: cluster 2023-07-10T16:58:25.910819+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:58:28.732 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:28 smithi192 bash[23730]: audit 2023-07-10T16:58:27.321388+0000 mgr.y (mgr.14144) 126 : audit [DBG] from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:58:28.733 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:28 smithi192 bash[23730]: cephadm 2023-07-10T16:58:27.326178+0000 mgr.y (mgr.14144) 127 : cephadm [INF] Applying service osd.None on host smithi192... 2023-07-10T16:58:28.733 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:28 smithi192 bash[23730]: cluster 2023-07-10T16:58:27.392820+0000 mgr.y (mgr.14144) 128 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:28.733 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:28 smithi192 bash[23730]: cluster 2023-07-10T16:58:27.426217+0000 mon.a (mon.0) 361 : cluster [INF] osd.4 [v2:172.21.15.192:6800/1383435807,v1:172.21.15.192:6801/1383435807] boot 2023-07-10T16:58:28.733 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:28 smithi192 bash[23730]: cluster 2023-07-10T16:58:27.426409+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2023-07-10T16:58:28.733 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:28 smithi192 bash[23730]: audit 2023-07-10T16:58:27.427264+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T16:58:28.733 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:28 smithi192 bash[23730]: cluster 2023-07-10T16:58:28.394597+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2023-07-10T16:58:28.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[23740]: cluster 2023-07-10T16:58:25.910645+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:58:28.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[23740]: cluster 2023-07-10T16:58:25.910819+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:58:28.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[23740]: audit 2023-07-10T16:58:27.321388+0000 mgr.y (mgr.14144) 126 : audit [DBG] from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:58:28.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[23740]: cephadm 2023-07-10T16:58:27.326178+0000 mgr.y (mgr.14144) 127 : cephadm [INF] Applying service osd.None on host smithi192... 2023-07-10T16:58:28.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[23740]: cluster 2023-07-10T16:58:27.392820+0000 mgr.y (mgr.14144) 128 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:28.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[23740]: cluster 2023-07-10T16:58:27.426217+0000 mon.a (mon.0) 361 : cluster [INF] osd.4 [v2:172.21.15.192:6800/1383435807,v1:172.21.15.192:6801/1383435807] boot 2023-07-10T16:58:28.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[23740]: cluster 2023-07-10T16:58:27.426409+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2023-07-10T16:58:28.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[23740]: audit 2023-07-10T16:58:27.427264+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T16:58:28.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[23740]: cluster 2023-07-10T16:58:28.394597+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2023-07-10T16:58:28.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[28351]: cluster 2023-07-10T16:58:25.910645+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:58:28.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[28351]: cluster 2023-07-10T16:58:25.910819+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:58:28.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[28351]: audit 2023-07-10T16:58:27.321388+0000 mgr.y (mgr.14144) 126 : audit [DBG] from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:58:28.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[28351]: cephadm 2023-07-10T16:58:27.326178+0000 mgr.y (mgr.14144) 127 : cephadm [INF] Applying service osd.None on host smithi192... 2023-07-10T16:58:28.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[28351]: cluster 2023-07-10T16:58:27.392820+0000 mgr.y (mgr.14144) 128 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2023-07-10T16:58:28.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[28351]: cluster 2023-07-10T16:58:27.426217+0000 mon.a (mon.0) 361 : cluster [INF] osd.4 [v2:172.21.15.192:6800/1383435807,v1:172.21.15.192:6801/1383435807] boot 2023-07-10T16:58:28.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[28351]: cluster 2023-07-10T16:58:27.426409+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2023-07-10T16:58:28.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[28351]: audit 2023-07-10T16:58:27.427264+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T16:58:28.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:28 smithi116 bash[28351]: cluster 2023-07-10T16:58:28.394597+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2023-07-10T16:58:29.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:29 smithi192 bash[23730]: cephadm 2023-07-10T16:58:28.668983+0000 mgr.y (mgr.14144) 129 : cephadm [INF] Detected new or changed devices on smithi192 2023-07-10T16:58:29.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:29 smithi192 bash[23730]: audit 2023-07-10T16:58:28.678309+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:29.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:29 smithi192 bash[23730]: audit 2023-07-10T16:58:28.684146+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:29.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:29 smithi192 bash[23730]: audit 2023-07-10T16:58:28.687761+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:58:29.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:29 smithi192 bash[23730]: audit 2023-07-10T16:58:28.696516+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:29.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:29 smithi192 bash[23730]: cluster 2023-07-10T16:58:29.405630+0000 mon.a (mon.0) 369 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2023-07-10T16:58:29.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:29 smithi116 bash[23740]: cephadm 2023-07-10T16:58:28.668983+0000 mgr.y (mgr.14144) 129 : cephadm [INF] Detected new or changed devices on smithi192 2023-07-10T16:58:29.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:29 smithi116 bash[23740]: audit 2023-07-10T16:58:28.678309+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:29.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:29 smithi116 bash[23740]: audit 2023-07-10T16:58:28.684146+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:29.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:29 smithi116 bash[23740]: audit 2023-07-10T16:58:28.687761+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:58:29.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:29 smithi116 bash[23740]: audit 2023-07-10T16:58:28.696516+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:29.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:29 smithi116 bash[23740]: cluster 2023-07-10T16:58:29.405630+0000 mon.a (mon.0) 369 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2023-07-10T16:58:29.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:29 smithi116 bash[28351]: cephadm 2023-07-10T16:58:28.668983+0000 mgr.y (mgr.14144) 129 : cephadm [INF] Detected new or changed devices on smithi192 2023-07-10T16:58:29.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:29 smithi116 bash[28351]: audit 2023-07-10T16:58:28.678309+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:29.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:29 smithi116 bash[28351]: audit 2023-07-10T16:58:28.684146+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:29.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:29 smithi116 bash[28351]: audit 2023-07-10T16:58:28.687761+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:58:29.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:29 smithi116 bash[28351]: audit 2023-07-10T16:58:28.696516+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:29.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:29 smithi116 bash[28351]: cluster 2023-07-10T16:58:29.405630+0000 mon.a (mon.0) 369 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2023-07-10T16:58:30.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:30 smithi192 bash[23730]: cluster 2023-07-10T16:58:29.393388+0000 mgr.y (mgr.14144) 130 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:30.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:30 smithi116 bash[23740]: cluster 2023-07-10T16:58:29.393388+0000 mgr.y (mgr.14144) 130 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:30.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:30 smithi116 bash[28351]: cluster 2023-07-10T16:58:29.393388+0000 mgr.y (mgr.14144) 130 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:31.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:31 smithi192 bash[23730]: audit 2023-07-10T16:58:31.265634+0000 mon.a (mon.0) 370 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14218aa2-8ec8-407f-879b-b4fc762177fc"}]: dispatch 2023-07-10T16:58:31.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:31 smithi192 bash[23730]: audit 2023-07-10T16:58:31.265764+0000 mon.b (mon.2) 11 : audit [INF] from='client.? 172.21.15.192:0/2260605893' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14218aa2-8ec8-407f-879b-b4fc762177fc"}]: dispatch 2023-07-10T16:58:31.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:31 smithi192 bash[23730]: audit 2023-07-10T16:58:31.273852+0000 mon.a (mon.0) 371 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14218aa2-8ec8-407f-879b-b4fc762177fc"}]': finished 2023-07-10T16:58:31.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:31 smithi192 bash[23730]: cluster 2023-07-10T16:58:31.274096+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2023-07-10T16:58:31.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:31 smithi192 bash[23730]: audit 2023-07-10T16:58:31.274404+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T16:58:31.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:31 smithi116 bash[23740]: audit 2023-07-10T16:58:31.265634+0000 mon.a (mon.0) 370 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14218aa2-8ec8-407f-879b-b4fc762177fc"}]: dispatch 2023-07-10T16:58:31.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:31 smithi116 bash[23740]: audit 2023-07-10T16:58:31.265764+0000 mon.b (mon.2) 11 : audit [INF] from='client.? 172.21.15.192:0/2260605893' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14218aa2-8ec8-407f-879b-b4fc762177fc"}]: dispatch 2023-07-10T16:58:31.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:31 smithi116 bash[23740]: audit 2023-07-10T16:58:31.273852+0000 mon.a (mon.0) 371 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14218aa2-8ec8-407f-879b-b4fc762177fc"}]': finished 2023-07-10T16:58:31.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:31 smithi116 bash[23740]: cluster 2023-07-10T16:58:31.274096+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2023-07-10T16:58:31.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:31 smithi116 bash[23740]: audit 2023-07-10T16:58:31.274404+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T16:58:31.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:31 smithi116 bash[28351]: audit 2023-07-10T16:58:31.265634+0000 mon.a (mon.0) 370 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14218aa2-8ec8-407f-879b-b4fc762177fc"}]: dispatch 2023-07-10T16:58:31.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:31 smithi116 bash[28351]: audit 2023-07-10T16:58:31.265764+0000 mon.b (mon.2) 11 : audit [INF] from='client.? 172.21.15.192:0/2260605893' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14218aa2-8ec8-407f-879b-b4fc762177fc"}]: dispatch 2023-07-10T16:58:31.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:31 smithi116 bash[28351]: audit 2023-07-10T16:58:31.273852+0000 mon.a (mon.0) 371 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14218aa2-8ec8-407f-879b-b4fc762177fc"}]': finished 2023-07-10T16:58:31.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:31 smithi116 bash[28351]: cluster 2023-07-10T16:58:31.274096+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2023-07-10T16:58:31.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:31 smithi116 bash[28351]: audit 2023-07-10T16:58:31.274404+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T16:58:32.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:32 smithi192 bash[23730]: cluster 2023-07-10T16:58:31.393928+0000 mgr.y (mgr.14144) 131 : cluster [DBG] pgmap v120: 1 pgs: 1 activating; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:32.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:32 smithi192 bash[23730]: audit 2023-07-10T16:58:31.942374+0000 mon.b (mon.2) 12 : audit [DBG] from='client.? 172.21.15.192:0/1505944485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:58:32.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:32 smithi116 bash[23740]: cluster 2023-07-10T16:58:31.393928+0000 mgr.y (mgr.14144) 131 : cluster [DBG] pgmap v120: 1 pgs: 1 activating; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:32.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:32 smithi116 bash[23740]: audit 2023-07-10T16:58:31.942374+0000 mon.b (mon.2) 12 : audit [DBG] from='client.? 172.21.15.192:0/1505944485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:58:32.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:32 smithi116 bash[28351]: cluster 2023-07-10T16:58:31.393928+0000 mgr.y (mgr.14144) 131 : cluster [DBG] pgmap v120: 1 pgs: 1 activating; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:32.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:32 smithi116 bash[28351]: audit 2023-07-10T16:58:31.942374+0000 mon.b (mon.2) 12 : audit [DBG] from='client.? 172.21.15.192:0/1505944485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:58:34.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:34 smithi192 bash[23730]: cluster 2023-07-10T16:58:33.394571+0000 mgr.y (mgr.14144) 132 : cluster [DBG] pgmap v121: 1 pgs: 1 activating; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:34.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:34 smithi116 bash[23740]: cluster 2023-07-10T16:58:33.394571+0000 mgr.y (mgr.14144) 132 : cluster [DBG] pgmap v121: 1 pgs: 1 activating; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:34.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:34 smithi116 bash[28351]: cluster 2023-07-10T16:58:33.394571+0000 mgr.y (mgr.14144) 132 : cluster [DBG] pgmap v121: 1 pgs: 1 activating; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:36.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:36 smithi192 bash[23730]: cluster 2023-07-10T16:58:35.395149+0000 mgr.y (mgr.14144) 133 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:36.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:36 smithi116 bash[23740]: cluster 2023-07-10T16:58:35.395149+0000 mgr.y (mgr.14144) 133 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:36.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:36 smithi116 bash[28351]: cluster 2023-07-10T16:58:35.395149+0000 mgr.y (mgr.14144) 133 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:38.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:38 smithi192 bash[23730]: cluster 2023-07-10T16:58:37.395679+0000 mgr.y (mgr.14144) 134 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:38.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:38 smithi116 bash[23740]: cluster 2023-07-10T16:58:37.395679+0000 mgr.y (mgr.14144) 134 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:38.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:38 smithi116 bash[28351]: cluster 2023-07-10T16:58:37.395679+0000 mgr.y (mgr.14144) 134 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:39.745 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:39 smithi192 bash[23730]: audit 2023-07-10T16:58:38.662429+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-10T16:58:39.745 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:39 smithi192 bash[23730]: audit 2023-07-10T16:58:38.664215+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:58:39.746 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:39 smithi192 bash[23730]: cephadm 2023-07-10T16:58:38.665721+0000 mgr.y (mgr.14144) 135 : cephadm [INF] Deploying daemon osd.5 on smithi192 2023-07-10T16:58:39.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:39 smithi116 bash[23740]: audit 2023-07-10T16:58:38.662429+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-10T16:58:39.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:39 smithi116 bash[23740]: audit 2023-07-10T16:58:38.664215+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:58:39.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:39 smithi116 bash[23740]: cephadm 2023-07-10T16:58:38.665721+0000 mgr.y (mgr.14144) 135 : cephadm [INF] Deploying daemon osd.5 on smithi192 2023-07-10T16:58:39.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:39 smithi116 bash[28351]: audit 2023-07-10T16:58:38.662429+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-10T16:58:39.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:39 smithi116 bash[28351]: audit 2023-07-10T16:58:38.664215+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:58:39.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:39 smithi116 bash[28351]: cephadm 2023-07-10T16:58:38.665721+0000 mgr.y (mgr.14144) 135 : cephadm [INF] Deploying daemon osd.5 on smithi192 2023-07-10T16:58:40.702 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:40 smithi192 bash[23730]: cluster 2023-07-10T16:58:39.396268+0000 mgr.y (mgr.14144) 136 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:40.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:40 smithi116 bash[28351]: cluster 2023-07-10T16:58:39.396268+0000 mgr.y (mgr.14144) 136 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:40.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:40 smithi116 bash[23740]: cluster 2023-07-10T16:58:39.396268+0000 mgr.y (mgr.14144) 136 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:40.962 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 5 on host 'smithi192' 2023-07-10T16:58:41.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:41 smithi192 bash[23730]: audit 2023-07-10T16:58:40.937447+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:58:41.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:41 smithi192 bash[23730]: audit 2023-07-10T16:58:40.945396+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:41.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:41 smithi192 bash[23730]: audit 2023-07-10T16:58:40.957750+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:41.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:41 smithi116 bash[23740]: audit 2023-07-10T16:58:40.937447+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:58:41.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:41 smithi116 bash[23740]: audit 2023-07-10T16:58:40.945396+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:41.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:41 smithi116 bash[23740]: audit 2023-07-10T16:58:40.957750+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:41.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:41 smithi116 bash[28351]: audit 2023-07-10T16:58:40.937447+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:58:41.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:41 smithi116 bash[28351]: audit 2023-07-10T16:58:40.945396+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:41.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:41 smithi116 bash[28351]: audit 2023-07-10T16:58:40.957750+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:41.985 DEBUG:teuthology.orchestra.run.smithi192:osd.5> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.5.service 2023-07-10T16:58:41.990 INFO:tasks.cephadm:Deploying osd.6 on smithi192 with /dev/vg_nvme/lv_2... 2023-07-10T16:58:41.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-10T16:58:42.612 INFO:journalctl@ceph.osd.5.smithi192.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T16:58:42.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:42 smithi192 bash[23730]: cluster 2023-07-10T16:58:41.396829+0000 mgr.y (mgr.14144) 137 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:42.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:42 smithi116 bash[23740]: cluster 2023-07-10T16:58:41.396829+0000 mgr.y (mgr.14144) 137 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:42.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:42 smithi116 bash[28351]: cluster 2023-07-10T16:58:41.396829+0000 mgr.y (mgr.14144) 137 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:43.106 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 16:58:42 smithi192 bash[29153]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-07-10T16:58:43.107 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 16:58:42 smithi192 bash[29153]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2023-07-10T16:58:43.107 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 16:58:42 smithi192 bash[29153]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2023-07-10T16:58:43.107 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 16:58:42 smithi192 bash[29153]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2023-07-10T16:58:43.107 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 16:58:42 smithi192 bash[29153]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2023-07-10T16:58:43.107 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 16:58:42 smithi192 bash[29153]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-07-10T16:58:43.108 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 16:58:42 smithi192 bash[29153]: --> ceph-volume lvm activate successful for osd ID: 5 2023-07-10T16:58:44.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:44 smithi192 bash[23730]: cluster 2023-07-10T16:58:43.397530+0000 mgr.y (mgr.14144) 138 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:44.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:44 smithi116 bash[23740]: cluster 2023-07-10T16:58:43.397530+0000 mgr.y (mgr.14144) 138 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:44.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:44 smithi116 bash[28351]: cluster 2023-07-10T16:58:43.397530+0000 mgr.y (mgr.14144) 138 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:45.886 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 16:58:45 smithi192 bash[29367]: debug 2023-07-10T16:58:45.668+0000 7fe6b38ba700 1 -- 172.21.15.192:0/417631567 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559361e484e0 con 0x559361ec1400 2023-07-10T16:58:45.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:45 smithi192 bash[23730]: audit 2023-07-10T16:58:44.715772+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:45.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:45 smithi192 bash[23730]: cluster 2023-07-10T16:58:45.398148+0000 mgr.y (mgr.14144) 139 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:46.142 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:45 smithi116 bash[23740]: audit 2023-07-10T16:58:44.715772+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:46.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:45 smithi116 bash[23740]: cluster 2023-07-10T16:58:45.398148+0000 mgr.y (mgr.14144) 139 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:46.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:45 smithi116 bash[28351]: audit 2023-07-10T16:58:44.715772+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:46.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:45 smithi116 bash[28351]: cluster 2023-07-10T16:58:45.398148+0000 mgr.y (mgr.14144) 139 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:46.548 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 16:58:46 smithi192 bash[29367]: debug 2023-07-10T16:58:46.200+0000 7fe6bb345080 -1 Falling back to public interface 2023-07-10T16:58:47.447 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-10T16:58:47.448 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-10T16:58:47.448 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2023-07-10T16:58:47.448 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2023-07-10T16:58:47.448 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0164734 s, 637 MB/s 2023-07-10T16:58:47.448 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2023-07-10T16:58:48.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi192:vg_nvme/lv_2 2023-07-10T16:58:48.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:48 smithi192 bash[23730]: cluster 2023-07-10T16:58:47.398710+0000 mgr.y (mgr.14144) 140 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:48.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:48 smithi116 bash[23740]: cluster 2023-07-10T16:58:47.398710+0000 mgr.y (mgr.14144) 140 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:48.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:48 smithi116 bash[28351]: cluster 2023-07-10T16:58:47.398710+0000 mgr.y (mgr.14144) 140 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:49.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:49 smithi192 bash[23730]: audit 2023-07-10T16:58:49.247755+0000 mon.b (mon.2) 13 : audit [INF] from='osd.5 [v2:172.21.15.192:6808/2450645600,v1:172.21.15.192:6809/2450645600]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-10T16:58:49.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:49 smithi192 bash[23730]: audit 2023-07-10T16:58:49.247888+0000 mon.a (mon.0) 380 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-10T16:58:49.631 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 16:58:49 smithi192 bash[29367]: debug 2023-07-10T16:58:49.240+0000 7fe6bb345080 -1 osd.5 0 log_to_monitors {default=true} 2023-07-10T16:58:49.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:49 smithi116 bash[23740]: audit 2023-07-10T16:58:49.247755+0000 mon.b (mon.2) 13 : audit [INF] from='osd.5 [v2:172.21.15.192:6808/2450645600,v1:172.21.15.192:6809/2450645600]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-10T16:58:49.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:49 smithi116 bash[23740]: audit 2023-07-10T16:58:49.247888+0000 mon.a (mon.0) 380 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-10T16:58:49.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:49 smithi116 bash[28351]: audit 2023-07-10T16:58:49.247755+0000 mon.b (mon.2) 13 : audit [INF] from='osd.5 [v2:172.21.15.192:6808/2450645600,v1:172.21.15.192:6809/2450645600]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-10T16:58:49.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:49 smithi116 bash[28351]: audit 2023-07-10T16:58:49.247888+0000 mon.a (mon.0) 380 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-10T16:58:50.812 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 16:58:50 smithi192 bash[29367]: debug 2023-07-10T16:58:50.480+0000 7fe6b18b6700 -1 osd.5 0 waiting for initial osdmap 2023-07-10T16:58:50.812 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:50 smithi192 bash[23730]: cluster 2023-07-10T16:58:49.399321+0000 mgr.y (mgr.14144) 141 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:50.812 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:50 smithi192 bash[23730]: audit 2023-07-10T16:58:49.469632+0000 mon.a (mon.0) 381 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-10T16:58:50.812 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:50 smithi192 bash[23730]: cluster 2023-07-10T16:58:49.469790+0000 mon.a (mon.0) 382 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2023-07-10T16:58:50.813 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:50 smithi192 bash[23730]: audit 2023-07-10T16:58:49.469997+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T16:58:50.813 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:50 smithi192 bash[23730]: audit 2023-07-10T16:58:49.471397+0000 mon.b (mon.2) 14 : audit [INF] from='osd.5 [v2:172.21.15.192:6808/2450645600,v1:172.21.15.192:6809/2450645600]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:58:50.813 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:50 smithi192 bash[23730]: audit 2023-07-10T16:58:49.471403+0000 mon.a (mon.0) 384 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:58:50.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:50 smithi116 bash[23740]: cluster 2023-07-10T16:58:49.399321+0000 mgr.y (mgr.14144) 141 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:50.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:50 smithi116 bash[23740]: audit 2023-07-10T16:58:49.469632+0000 mon.a (mon.0) 381 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-10T16:58:50.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:50 smithi116 bash[23740]: cluster 2023-07-10T16:58:49.469790+0000 mon.a (mon.0) 382 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2023-07-10T16:58:50.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:50 smithi116 bash[23740]: audit 2023-07-10T16:58:49.469997+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T16:58:50.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:50 smithi116 bash[23740]: audit 2023-07-10T16:58:49.471397+0000 mon.b (mon.2) 14 : audit [INF] from='osd.5 [v2:172.21.15.192:6808/2450645600,v1:172.21.15.192:6809/2450645600]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:58:50.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:50 smithi116 bash[23740]: audit 2023-07-10T16:58:49.471403+0000 mon.a (mon.0) 384 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:58:50.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:50 smithi116 bash[28351]: cluster 2023-07-10T16:58:49.399321+0000 mgr.y (mgr.14144) 141 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:50.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:50 smithi116 bash[28351]: audit 2023-07-10T16:58:49.469632+0000 mon.a (mon.0) 381 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-10T16:58:50.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:50 smithi116 bash[28351]: cluster 2023-07-10T16:58:49.469790+0000 mon.a (mon.0) 382 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2023-07-10T16:58:50.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:50 smithi116 bash[28351]: audit 2023-07-10T16:58:49.469997+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T16:58:50.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:50 smithi116 bash[28351]: audit 2023-07-10T16:58:49.471397+0000 mon.b (mon.2) 14 : audit [INF] from='osd.5 [v2:172.21.15.192:6808/2450645600,v1:172.21.15.192:6809/2450645600]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:58:50.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:50 smithi116 bash[28351]: audit 2023-07-10T16:58:49.471403+0000 mon.a (mon.0) 384 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:58:51.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:51 smithi192 bash[23730]: audit 2023-07-10T16:58:50.474471+0000 mon.a (mon.0) 385 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-10T16:58:51.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:51 smithi192 bash[23730]: cluster 2023-07-10T16:58:50.474659+0000 mon.a (mon.0) 386 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2023-07-10T16:58:51.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:51 smithi192 bash[23730]: audit 2023-07-10T16:58:50.475538+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T16:58:51.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:51 smithi192 bash[23730]: audit 2023-07-10T16:58:50.481817+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T16:58:51.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:51 smithi192 bash[23730]: audit 2023-07-10T16:58:51.330716+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:51.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:51 smithi192 bash[23730]: audit 2023-07-10T16:58:51.338887+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:51.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:51 smithi192 bash[23730]: audit 2023-07-10T16:58:51.341229+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:58:51.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:51 smithi192 bash[23730]: audit 2023-07-10T16:58:51.349224+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:51.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:51 smithi116 bash[28351]: audit 2023-07-10T16:58:50.474471+0000 mon.a (mon.0) 385 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-10T16:58:51.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:51 smithi116 bash[28351]: cluster 2023-07-10T16:58:50.474659+0000 mon.a (mon.0) 386 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2023-07-10T16:58:51.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:51 smithi116 bash[28351]: audit 2023-07-10T16:58:50.475538+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T16:58:51.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:51 smithi116 bash[28351]: audit 2023-07-10T16:58:50.481817+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T16:58:51.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:51 smithi116 bash[28351]: audit 2023-07-10T16:58:51.330716+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:51.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:51 smithi116 bash[28351]: audit 2023-07-10T16:58:51.338887+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:51.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:51 smithi116 bash[28351]: audit 2023-07-10T16:58:51.341229+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:58:51.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:51 smithi116 bash[28351]: audit 2023-07-10T16:58:51.349224+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:51.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:51 smithi116 bash[23740]: audit 2023-07-10T16:58:50.474471+0000 mon.a (mon.0) 385 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-10T16:58:51.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:51 smithi116 bash[23740]: cluster 2023-07-10T16:58:50.474659+0000 mon.a (mon.0) 386 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2023-07-10T16:58:51.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:51 smithi116 bash[23740]: audit 2023-07-10T16:58:50.475538+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T16:58:51.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:51 smithi116 bash[23740]: audit 2023-07-10T16:58:50.481817+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T16:58:51.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:51 smithi116 bash[23740]: audit 2023-07-10T16:58:51.330716+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:51.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:51 smithi116 bash[23740]: audit 2023-07-10T16:58:51.338887+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:51.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:51 smithi116 bash[23740]: audit 2023-07-10T16:58:51.341229+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:58:51.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:51 smithi116 bash[23740]: audit 2023-07-10T16:58:51.349224+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:58:52.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:52 smithi192 bash[23730]: cluster 2023-07-10T16:58:50.215353+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:58:52.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:52 smithi192 bash[23730]: cluster 2023-07-10T16:58:50.215517+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:58:52.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:52 smithi192 bash[23730]: cephadm 2023-07-10T16:58:51.321697+0000 mgr.y (mgr.14144) 142 : cephadm [INF] Detected new or changed devices on smithi192 2023-07-10T16:58:52.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:52 smithi192 bash[23730]: cluster 2023-07-10T16:58:51.399841+0000 mgr.y (mgr.14144) 143 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:52.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:52 smithi192 bash[23730]: cluster 2023-07-10T16:58:51.485640+0000 mon.a (mon.0) 393 : cluster [INF] osd.5 [v2:172.21.15.192:6808/2450645600,v1:172.21.15.192:6809/2450645600] boot 2023-07-10T16:58:52.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:52 smithi192 bash[23730]: cluster 2023-07-10T16:58:51.485926+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2023-07-10T16:58:52.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:52 smithi192 bash[23730]: audit 2023-07-10T16:58:51.486822+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T16:58:52.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:52 smithi192 bash[23730]: audit 2023-07-10T16:58:52.070338+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:58:52.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:52 smithi192 bash[23730]: audit 2023-07-10T16:58:52.074071+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:58:52.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:52 smithi192 bash[23730]: audit 2023-07-10T16:58:52.075894+0000 mon.a (mon.0) 398 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:58:52.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[23740]: cluster 2023-07-10T16:58:50.215353+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:58:52.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[23740]: cluster 2023-07-10T16:58:50.215517+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:58:52.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[23740]: cephadm 2023-07-10T16:58:51.321697+0000 mgr.y (mgr.14144) 142 : cephadm [INF] Detected new or changed devices on smithi192 2023-07-10T16:58:52.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[23740]: cluster 2023-07-10T16:58:51.399841+0000 mgr.y (mgr.14144) 143 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:52.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[23740]: cluster 2023-07-10T16:58:51.485640+0000 mon.a (mon.0) 393 : cluster [INF] osd.5 [v2:172.21.15.192:6808/2450645600,v1:172.21.15.192:6809/2450645600] boot 2023-07-10T16:58:52.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[23740]: cluster 2023-07-10T16:58:51.485926+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2023-07-10T16:58:52.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[23740]: audit 2023-07-10T16:58:51.486822+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T16:58:52.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[23740]: audit 2023-07-10T16:58:52.070338+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:58:52.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[23740]: audit 2023-07-10T16:58:52.074071+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:58:52.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[23740]: audit 2023-07-10T16:58:52.075894+0000 mon.a (mon.0) 398 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:58:52.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[28351]: cluster 2023-07-10T16:58:50.215353+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:58:52.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[28351]: cluster 2023-07-10T16:58:50.215517+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:58:52.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[28351]: cephadm 2023-07-10T16:58:51.321697+0000 mgr.y (mgr.14144) 142 : cephadm [INF] Detected new or changed devices on smithi192 2023-07-10T16:58:52.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[28351]: cluster 2023-07-10T16:58:51.399841+0000 mgr.y (mgr.14144) 143 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2023-07-10T16:58:52.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[28351]: cluster 2023-07-10T16:58:51.485640+0000 mon.a (mon.0) 393 : cluster [INF] osd.5 [v2:172.21.15.192:6808/2450645600,v1:172.21.15.192:6809/2450645600] boot 2023-07-10T16:58:52.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[28351]: cluster 2023-07-10T16:58:51.485926+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2023-07-10T16:58:52.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[28351]: audit 2023-07-10T16:58:51.486822+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T16:58:52.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[28351]: audit 2023-07-10T16:58:52.070338+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:58:52.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[28351]: audit 2023-07-10T16:58:52.074071+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:58:52.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:52 smithi116 bash[28351]: audit 2023-07-10T16:58:52.075894+0000 mon.a (mon.0) 398 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:58:53.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:53 smithi192 bash[23730]: audit 2023-07-10T16:58:52.068647+0000 mgr.y (mgr.14144) 144 : audit [DBG] from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:58:53.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:53 smithi192 bash[23730]: cephadm 2023-07-10T16:58:52.073498+0000 mgr.y (mgr.14144) 145 : cephadm [INF] Applying service osd.None on host smithi192... 2023-07-10T16:58:53.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:53 smithi192 bash[23730]: cluster 2023-07-10T16:58:52.497964+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2023-07-10T16:58:53.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:53 smithi116 bash[28351]: audit 2023-07-10T16:58:52.068647+0000 mgr.y (mgr.14144) 144 : audit [DBG] from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:58:53.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:53 smithi116 bash[28351]: cephadm 2023-07-10T16:58:52.073498+0000 mgr.y (mgr.14144) 145 : cephadm [INF] Applying service osd.None on host smithi192... 2023-07-10T16:58:53.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:53 smithi116 bash[28351]: cluster 2023-07-10T16:58:52.497964+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2023-07-10T16:58:53.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:53 smithi116 bash[23740]: audit 2023-07-10T16:58:52.068647+0000 mgr.y (mgr.14144) 144 : audit [DBG] from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:58:53.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:53 smithi116 bash[23740]: cephadm 2023-07-10T16:58:52.073498+0000 mgr.y (mgr.14144) 145 : cephadm [INF] Applying service osd.None on host smithi192... 2023-07-10T16:58:53.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:53 smithi116 bash[23740]: cluster 2023-07-10T16:58:52.497964+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2023-07-10T16:58:54.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:54 smithi192 bash[23730]: cluster 2023-07-10T16:58:53.400502+0000 mgr.y (mgr.14144) 146 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:58:54.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:54 smithi192 bash[23730]: cluster 2023-07-10T16:58:53.513363+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2023-07-10T16:58:54.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:54 smithi116 bash[23740]: cluster 2023-07-10T16:58:53.400502+0000 mgr.y (mgr.14144) 146 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:58:54.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:54 smithi116 bash[23740]: cluster 2023-07-10T16:58:53.513363+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2023-07-10T16:58:54.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:54 smithi116 bash[28351]: cluster 2023-07-10T16:58:53.400502+0000 mgr.y (mgr.14144) 146 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:58:54.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:54 smithi116 bash[28351]: cluster 2023-07-10T16:58:53.513363+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2023-07-10T16:58:56.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:56 smithi192 bash[23730]: cluster 2023-07-10T16:58:55.401109+0000 mgr.y (mgr.14144) 147 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:58:56.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:56 smithi192 bash[23730]: audit 2023-07-10T16:58:55.759408+0000 mon.b (mon.2) 15 : audit [INF] from='client.? 172.21.15.192:0/2335927971' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b279a6e4-a920-46c2-b19b-59131f5c68f9"}]: dispatch 2023-07-10T16:58:56.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:56 smithi192 bash[23730]: audit 2023-07-10T16:58:55.759942+0000 mon.a (mon.0) 401 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b279a6e4-a920-46c2-b19b-59131f5c68f9"}]: dispatch 2023-07-10T16:58:56.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:56 smithi192 bash[23730]: audit 2023-07-10T16:58:55.770766+0000 mon.a (mon.0) 402 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b279a6e4-a920-46c2-b19b-59131f5c68f9"}]': finished 2023-07-10T16:58:56.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:56 smithi192 bash[23730]: cluster 2023-07-10T16:58:55.770991+0000 mon.a (mon.0) 403 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2023-07-10T16:58:56.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:56 smithi192 bash[23730]: audit 2023-07-10T16:58:55.771333+0000 mon.a (mon.0) 404 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T16:58:56.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:56 smithi192 bash[23730]: audit 2023-07-10T16:58:56.400033+0000 mon.b (mon.2) 16 : audit [DBG] from='client.? 172.21.15.192:0/832264297' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:58:56.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:56 smithi116 bash[23740]: cluster 2023-07-10T16:58:55.401109+0000 mgr.y (mgr.14144) 147 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:58:56.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:56 smithi116 bash[23740]: audit 2023-07-10T16:58:55.759408+0000 mon.b (mon.2) 15 : audit [INF] from='client.? 172.21.15.192:0/2335927971' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b279a6e4-a920-46c2-b19b-59131f5c68f9"}]: dispatch 2023-07-10T16:58:56.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:56 smithi116 bash[23740]: audit 2023-07-10T16:58:55.759942+0000 mon.a (mon.0) 401 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b279a6e4-a920-46c2-b19b-59131f5c68f9"}]: dispatch 2023-07-10T16:58:56.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:56 smithi116 bash[23740]: audit 2023-07-10T16:58:55.770766+0000 mon.a (mon.0) 402 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b279a6e4-a920-46c2-b19b-59131f5c68f9"}]': finished 2023-07-10T16:58:56.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:56 smithi116 bash[23740]: cluster 2023-07-10T16:58:55.770991+0000 mon.a (mon.0) 403 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2023-07-10T16:58:56.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:56 smithi116 bash[23740]: audit 2023-07-10T16:58:55.771333+0000 mon.a (mon.0) 404 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T16:58:56.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:56 smithi116 bash[23740]: audit 2023-07-10T16:58:56.400033+0000 mon.b (mon.2) 16 : audit [DBG] from='client.? 172.21.15.192:0/832264297' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:58:56.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:56 smithi116 bash[28351]: cluster 2023-07-10T16:58:55.401109+0000 mgr.y (mgr.14144) 147 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:58:56.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:56 smithi116 bash[28351]: audit 2023-07-10T16:58:55.759408+0000 mon.b (mon.2) 15 : audit [INF] from='client.? 172.21.15.192:0/2335927971' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b279a6e4-a920-46c2-b19b-59131f5c68f9"}]: dispatch 2023-07-10T16:58:56.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:56 smithi116 bash[28351]: audit 2023-07-10T16:58:55.759942+0000 mon.a (mon.0) 401 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b279a6e4-a920-46c2-b19b-59131f5c68f9"}]: dispatch 2023-07-10T16:58:56.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:56 smithi116 bash[28351]: audit 2023-07-10T16:58:55.770766+0000 mon.a (mon.0) 402 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b279a6e4-a920-46c2-b19b-59131f5c68f9"}]': finished 2023-07-10T16:58:56.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:56 smithi116 bash[28351]: cluster 2023-07-10T16:58:55.770991+0000 mon.a (mon.0) 403 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2023-07-10T16:58:56.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:56 smithi116 bash[28351]: audit 2023-07-10T16:58:55.771333+0000 mon.a (mon.0) 404 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T16:58:56.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:56 smithi116 bash[28351]: audit 2023-07-10T16:58:56.400033+0000 mon.b (mon.2) 16 : audit [DBG] from='client.? 172.21.15.192:0/832264297' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:58:58.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:58:58 smithi192 bash[23730]: cluster 2023-07-10T16:58:57.401687+0000 mgr.y (mgr.14144) 148 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:58:58.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:58:58 smithi116 bash[23740]: cluster 2023-07-10T16:58:57.401687+0000 mgr.y (mgr.14144) 148 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:58:58.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:58:58 smithi116 bash[28351]: cluster 2023-07-10T16:58:57.401687+0000 mgr.y (mgr.14144) 148 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:00.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:00 smithi192 bash[23730]: cluster 2023-07-10T16:58:59.402313+0000 mgr.y (mgr.14144) 149 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:00.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:00 smithi116 bash[23740]: cluster 2023-07-10T16:58:59.402313+0000 mgr.y (mgr.14144) 149 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:00.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:00 smithi116 bash[28351]: cluster 2023-07-10T16:58:59.402313+0000 mgr.y (mgr.14144) 149 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:02.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:02 smithi192 bash[23730]: cluster 2023-07-10T16:59:01.402875+0000 mgr.y (mgr.14144) 150 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:02.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:02 smithi116 bash[23740]: cluster 2023-07-10T16:59:01.402875+0000 mgr.y (mgr.14144) 150 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:02.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:02 smithi116 bash[28351]: cluster 2023-07-10T16:59:01.402875+0000 mgr.y (mgr.14144) 150 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:03.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:03 smithi192 bash[23730]: audit 2023-07-10T16:59:03.128675+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-10T16:59:03.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:03 smithi192 bash[23730]: audit 2023-07-10T16:59:03.130579+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:59:03.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:03 smithi116 bash[23740]: audit 2023-07-10T16:59:03.128675+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-10T16:59:03.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:03 smithi116 bash[23740]: audit 2023-07-10T16:59:03.130579+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:59:03.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:03 smithi116 bash[28351]: audit 2023-07-10T16:59:03.128675+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-10T16:59:03.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:03 smithi116 bash[28351]: audit 2023-07-10T16:59:03.130579+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:59:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:04 smithi192 bash[23730]: cephadm 2023-07-10T16:59:03.132255+0000 mgr.y (mgr.14144) 151 : cephadm [INF] Deploying daemon osd.6 on smithi192 2023-07-10T16:59:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:04 smithi192 bash[23730]: cluster 2023-07-10T16:59:03.403617+0000 mgr.y (mgr.14144) 152 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:04.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:04 smithi116 bash[23740]: cephadm 2023-07-10T16:59:03.132255+0000 mgr.y (mgr.14144) 151 : cephadm [INF] Deploying daemon osd.6 on smithi192 2023-07-10T16:59:04.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:04 smithi116 bash[23740]: cluster 2023-07-10T16:59:03.403617+0000 mgr.y (mgr.14144) 152 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:04.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:04 smithi116 bash[28351]: cephadm 2023-07-10T16:59:03.132255+0000 mgr.y (mgr.14144) 151 : cephadm [INF] Deploying daemon osd.6 on smithi192 2023-07-10T16:59:04.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:04 smithi116 bash[28351]: cluster 2023-07-10T16:59:03.403617+0000 mgr.y (mgr.14144) 152 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:05.402 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 6 on host 'smithi192' 2023-07-10T16:59:05.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:05 smithi192 bash[23730]: audit 2023-07-10T16:59:05.377684+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:59:05.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:05 smithi192 bash[23730]: audit 2023-07-10T16:59:05.386677+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:05.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:05 smithi192 bash[23730]: audit 2023-07-10T16:59:05.399375+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:05.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:05 smithi116 bash[23740]: audit 2023-07-10T16:59:05.377684+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:59:05.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:05 smithi116 bash[23740]: audit 2023-07-10T16:59:05.386677+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:05.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:05 smithi116 bash[23740]: audit 2023-07-10T16:59:05.399375+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:05.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:05 smithi116 bash[28351]: audit 2023-07-10T16:59:05.377684+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:59:05.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:05 smithi116 bash[28351]: audit 2023-07-10T16:59:05.386677+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:05.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:05 smithi116 bash[28351]: audit 2023-07-10T16:59:05.399375+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:06.562 DEBUG:teuthology.orchestra.run.smithi192:osd.6> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.6.service 2023-07-10T16:59:06.565 INFO:tasks.cephadm:Deploying osd.7 on smithi192 with /dev/vg_nvme/lv_1... 2023-07-10T16:59:06.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-10T16:59:06.760 INFO:journalctl@ceph.osd.6.smithi192.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T16:59:06.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:06 smithi192 bash[23730]: cluster 2023-07-10T16:59:05.404173+0000 mgr.y (mgr.14144) 153 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:06.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:06 smithi116 bash[23740]: cluster 2023-07-10T16:59:05.404173+0000 mgr.y (mgr.14144) 153 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:06.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:06 smithi116 bash[28351]: cluster 2023-07-10T16:59:05.404173+0000 mgr.y (mgr.14144) 153 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:07.886 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 16:59:07 smithi192 bash[31913]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-07-10T16:59:07.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 16:59:07 smithi192 bash[31913]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-6 --no-mon-config 2023-07-10T16:59:07.888 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 16:59:07 smithi192 bash[31913]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6/block 2023-07-10T16:59:07.888 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 16:59:07 smithi192 bash[31913]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-6/block 2023-07-10T16:59:07.888 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 16:59:07 smithi192 bash[31913]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2023-07-10T16:59:07.889 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 16:59:07 smithi192 bash[31913]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-07-10T16:59:07.889 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 16:59:07 smithi192 bash[31913]: --> ceph-volume lvm activate successful for osd ID: 6 2023-07-10T16:59:08.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:08 smithi192 bash[23730]: cluster 2023-07-10T16:59:07.404793+0000 mgr.y (mgr.14144) 154 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:08.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:08 smithi116 bash[28351]: cluster 2023-07-10T16:59:07.404793+0000 mgr.y (mgr.14144) 154 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:08.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:08 smithi116 bash[23740]: cluster 2023-07-10T16:59:07.404793+0000 mgr.y (mgr.14144) 154 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:10.136 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 16:59:10 smithi192 bash[32145]: debug 2023-07-10T16:59:10.048+0000 7f9aa27cd700 1 -- 172.21.15.192:0/207925061 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e684b1a4e0 con 0x55e684b92c00 2023-07-10T16:59:10.636 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 16:59:10 smithi192 bash[32145]: debug 2023-07-10T16:59:10.604+0000 7f9aaa258080 -1 Falling back to public interface 2023-07-10T16:59:10.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:10 smithi192 bash[23730]: cluster 2023-07-10T16:59:09.405400+0000 mgr.y (mgr.14144) 155 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:10.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:10 smithi192 bash[23730]: audit 2023-07-10T16:59:10.192590+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:10.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:10 smithi116 bash[28351]: cluster 2023-07-10T16:59:09.405400+0000 mgr.y (mgr.14144) 155 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:10.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:10 smithi116 bash[28351]: audit 2023-07-10T16:59:10.192590+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:10.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:10 smithi116 bash[23740]: cluster 2023-07-10T16:59:09.405400+0000 mgr.y (mgr.14144) 155 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:10.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:10 smithi116 bash[23740]: audit 2023-07-10T16:59:10.192590+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:10.952 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-10T16:59:10.952 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-10T16:59:10.952 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2023-07-10T16:59:10.953 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2023-07-10T16:59:10.953 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144506 s, 726 MB/s 2023-07-10T16:59:10.953 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2023-07-10T16:59:11.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi192:vg_nvme/lv_1 2023-07-10T16:59:12.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:12 smithi192 bash[23730]: cluster 2023-07-10T16:59:11.405967+0000 mgr.y (mgr.14144) 156 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:12.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:12 smithi116 bash[23740]: cluster 2023-07-10T16:59:11.405967+0000 mgr.y (mgr.14144) 156 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:12.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:12 smithi116 bash[28351]: cluster 2023-07-10T16:59:11.405967+0000 mgr.y (mgr.14144) 156 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:13.925 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 16:59:13 smithi192 bash[32145]: debug 2023-07-10T16:59:13.647+0000 7f9aaa258080 -1 osd.6 0 log_to_monitors {default=true} 2023-07-10T16:59:14.543 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:14 smithi192 bash[23730]: cluster 2023-07-10T16:59:13.406649+0000 mgr.y (mgr.14144) 157 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:14.543 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:14 smithi192 bash[23730]: audit 2023-07-10T16:59:13.655366+0000 mon.a (mon.0) 411 : audit [INF] from='osd.6 [v2:172.21.15.192:6816/425451514,v1:172.21.15.192:6817/425451514]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-10T16:59:14.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:14 smithi116 bash[23740]: cluster 2023-07-10T16:59:13.406649+0000 mgr.y (mgr.14144) 157 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:14.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:14 smithi116 bash[23740]: audit 2023-07-10T16:59:13.655366+0000 mon.a (mon.0) 411 : audit [INF] from='osd.6 [v2:172.21.15.192:6816/425451514,v1:172.21.15.192:6817/425451514]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-10T16:59:14.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:14 smithi116 bash[28351]: cluster 2023-07-10T16:59:13.406649+0000 mgr.y (mgr.14144) 157 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:14.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:14 smithi116 bash[28351]: audit 2023-07-10T16:59:13.655366+0000 mon.a (mon.0) 411 : audit [INF] from='osd.6 [v2:172.21.15.192:6816/425451514,v1:172.21.15.192:6817/425451514]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-10T16:59:15.685 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 16:59:15 smithi192 bash[32145]: debug 2023-07-10T16:59:15.427+0000 7f9aa07c9700 -1 osd.6 0 waiting for initial osdmap 2023-07-10T16:59:15.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:15 smithi192 bash[23730]: audit 2023-07-10T16:59:14.419298+0000 mon.a (mon.0) 412 : audit [INF] from='osd.6 [v2:172.21.15.192:6816/425451514,v1:172.21.15.192:6817/425451514]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-10T16:59:15.686 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:15 smithi192 bash[23730]: cluster 2023-07-10T16:59:14.419477+0000 mon.a (mon.0) 413 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2023-07-10T16:59:15.686 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:15 smithi192 bash[23730]: audit 2023-07-10T16:59:14.419692+0000 mon.a (mon.0) 414 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T16:59:15.686 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:15 smithi192 bash[23730]: audit 2023-07-10T16:59:14.420528+0000 mon.a (mon.0) 415 : audit [INF] from='osd.6 [v2:172.21.15.192:6816/425451514,v1:172.21.15.192:6817/425451514]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:59:15.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:15 smithi116 bash[23740]: audit 2023-07-10T16:59:14.419298+0000 mon.a (mon.0) 412 : audit [INF] from='osd.6 [v2:172.21.15.192:6816/425451514,v1:172.21.15.192:6817/425451514]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-10T16:59:15.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:15 smithi116 bash[23740]: cluster 2023-07-10T16:59:14.419477+0000 mon.a (mon.0) 413 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2023-07-10T16:59:15.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:15 smithi116 bash[23740]: audit 2023-07-10T16:59:14.419692+0000 mon.a (mon.0) 414 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T16:59:15.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:15 smithi116 bash[23740]: audit 2023-07-10T16:59:14.420528+0000 mon.a (mon.0) 415 : audit [INF] from='osd.6 [v2:172.21.15.192:6816/425451514,v1:172.21.15.192:6817/425451514]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:59:15.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:15 smithi116 bash[28351]: audit 2023-07-10T16:59:14.419298+0000 mon.a (mon.0) 412 : audit [INF] from='osd.6 [v2:172.21.15.192:6816/425451514,v1:172.21.15.192:6817/425451514]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-10T16:59:15.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:15 smithi116 bash[28351]: cluster 2023-07-10T16:59:14.419477+0000 mon.a (mon.0) 413 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2023-07-10T16:59:15.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:15 smithi116 bash[28351]: audit 2023-07-10T16:59:14.419692+0000 mon.a (mon.0) 414 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T16:59:15.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:15 smithi116 bash[28351]: audit 2023-07-10T16:59:14.420528+0000 mon.a (mon.0) 415 : audit [INF] from='osd.6 [v2:172.21.15.192:6816/425451514,v1:172.21.15.192:6817/425451514]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:59:16.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:16 smithi192 bash[23730]: cluster 2023-07-10T16:59:15.407277+0000 mgr.y (mgr.14144) 158 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:16.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:16 smithi192 bash[23730]: audit 2023-07-10T16:59:15.423982+0000 mon.a (mon.0) 416 : audit [INF] from='osd.6 [v2:172.21.15.192:6816/425451514,v1:172.21.15.192:6817/425451514]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-10T16:59:16.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:16 smithi192 bash[23730]: cluster 2023-07-10T16:59:15.424357+0000 mon.a (mon.0) 417 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2023-07-10T16:59:16.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:16 smithi192 bash[23730]: audit 2023-07-10T16:59:15.425422+0000 mon.a (mon.0) 418 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T16:59:16.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:16 smithi192 bash[23730]: audit 2023-07-10T16:59:15.433573+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T16:59:16.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:16 smithi192 bash[23730]: audit 2023-07-10T16:59:16.033330+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:59:16.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:16 smithi192 bash[23730]: audit 2023-07-10T16:59:16.036934+0000 mon.a (mon.0) 421 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:59:16.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:16 smithi192 bash[23730]: audit 2023-07-10T16:59:16.038759+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:59:16.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:16 smithi192 bash[23730]: cluster 2023-07-10T16:59:16.425109+0000 mon.a (mon.0) 423 : cluster [INF] osd.6 [v2:172.21.15.192:6816/425451514,v1:172.21.15.192:6817/425451514] boot 2023-07-10T16:59:16.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:16 smithi192 bash[23730]: cluster 2023-07-10T16:59:16.425193+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2023-07-10T16:59:16.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:16 smithi192 bash[23730]: audit 2023-07-10T16:59:16.426194+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T16:59:16.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[23740]: cluster 2023-07-10T16:59:15.407277+0000 mgr.y (mgr.14144) 158 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:16.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[23740]: audit 2023-07-10T16:59:15.423982+0000 mon.a (mon.0) 416 : audit [INF] from='osd.6 [v2:172.21.15.192:6816/425451514,v1:172.21.15.192:6817/425451514]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-10T16:59:16.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[23740]: cluster 2023-07-10T16:59:15.424357+0000 mon.a (mon.0) 417 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2023-07-10T16:59:16.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[23740]: audit 2023-07-10T16:59:15.425422+0000 mon.a (mon.0) 418 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T16:59:16.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[23740]: audit 2023-07-10T16:59:15.433573+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T16:59:16.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[23740]: audit 2023-07-10T16:59:16.033330+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:59:16.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[23740]: audit 2023-07-10T16:59:16.036934+0000 mon.a (mon.0) 421 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:59:16.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[23740]: audit 2023-07-10T16:59:16.038759+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:59:16.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[23740]: cluster 2023-07-10T16:59:16.425109+0000 mon.a (mon.0) 423 : cluster [INF] osd.6 [v2:172.21.15.192:6816/425451514,v1:172.21.15.192:6817/425451514] boot 2023-07-10T16:59:16.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[23740]: cluster 2023-07-10T16:59:16.425193+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2023-07-10T16:59:16.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[23740]: audit 2023-07-10T16:59:16.426194+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T16:59:16.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[28351]: cluster 2023-07-10T16:59:15.407277+0000 mgr.y (mgr.14144) 158 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2023-07-10T16:59:16.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[28351]: audit 2023-07-10T16:59:15.423982+0000 mon.a (mon.0) 416 : audit [INF] from='osd.6 [v2:172.21.15.192:6816/425451514,v1:172.21.15.192:6817/425451514]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-10T16:59:16.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[28351]: cluster 2023-07-10T16:59:15.424357+0000 mon.a (mon.0) 417 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2023-07-10T16:59:16.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[28351]: audit 2023-07-10T16:59:15.425422+0000 mon.a (mon.0) 418 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T16:59:16.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[28351]: audit 2023-07-10T16:59:15.433573+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T16:59:16.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[28351]: audit 2023-07-10T16:59:16.033330+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-10T16:59:16.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[28351]: audit 2023-07-10T16:59:16.036934+0000 mon.a (mon.0) 421 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-10T16:59:16.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[28351]: audit 2023-07-10T16:59:16.038759+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:59:16.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[28351]: cluster 2023-07-10T16:59:16.425109+0000 mon.a (mon.0) 423 : cluster [INF] osd.6 [v2:172.21.15.192:6816/425451514,v1:172.21.15.192:6817/425451514] boot 2023-07-10T16:59:16.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[28351]: cluster 2023-07-10T16:59:16.425193+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2023-07-10T16:59:16.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:16 smithi116 bash[28351]: audit 2023-07-10T16:59:16.426194+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T16:59:17.726 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:17 smithi192 bash[23730]: cluster 2023-07-10T16:59:14.648418+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:59:17.727 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:17 smithi192 bash[23730]: cluster 2023-07-10T16:59:14.648613+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:59:17.727 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:17 smithi192 bash[23730]: audit 2023-07-10T16:59:16.031627+0000 mgr.y (mgr.14144) 159 : audit [DBG] from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:59:17.727 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:17 smithi192 bash[23730]: cephadm 2023-07-10T16:59:16.036404+0000 mgr.y (mgr.14144) 160 : cephadm [INF] Applying service osd.None on host smithi192... 2023-07-10T16:59:17.727 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:17 smithi192 bash[23730]: audit 2023-07-10T16:59:17.259300+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:17.727 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:17 smithi192 bash[23730]: cluster 2023-07-10T16:59:17.427193+0000 mon.a (mon.0) 427 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2023-07-10T16:59:17.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:17 smithi116 bash[23740]: cluster 2023-07-10T16:59:14.648418+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:59:17.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:17 smithi116 bash[23740]: cluster 2023-07-10T16:59:14.648613+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:59:17.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:17 smithi116 bash[23740]: audit 2023-07-10T16:59:16.031627+0000 mgr.y (mgr.14144) 159 : audit [DBG] from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:59:17.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:17 smithi116 bash[23740]: cephadm 2023-07-10T16:59:16.036404+0000 mgr.y (mgr.14144) 160 : cephadm [INF] Applying service osd.None on host smithi192... 2023-07-10T16:59:17.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:17 smithi116 bash[23740]: audit 2023-07-10T16:59:17.259300+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:17.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:17 smithi116 bash[23740]: cluster 2023-07-10T16:59:17.427193+0000 mon.a (mon.0) 427 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2023-07-10T16:59:17.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:17 smithi116 bash[28351]: cluster 2023-07-10T16:59:14.648418+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:59:17.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:17 smithi116 bash[28351]: cluster 2023-07-10T16:59:14.648613+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:59:17.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:17 smithi116 bash[28351]: audit 2023-07-10T16:59:16.031627+0000 mgr.y (mgr.14144) 159 : audit [DBG] from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T16:59:17.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:17 smithi116 bash[28351]: cephadm 2023-07-10T16:59:16.036404+0000 mgr.y (mgr.14144) 160 : cephadm [INF] Applying service osd.None on host smithi192... 2023-07-10T16:59:17.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:17 smithi116 bash[28351]: audit 2023-07-10T16:59:17.259300+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:17.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:17 smithi116 bash[28351]: cluster 2023-07-10T16:59:17.427193+0000 mon.a (mon.0) 427 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2023-07-10T16:59:18.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:18 smithi192 bash[23730]: cephadm 2023-07-10T16:59:17.249631+0000 mgr.y (mgr.14144) 161 : cephadm [INF] Detected new or changed devices on smithi192 2023-07-10T16:59:18.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:18 smithi192 bash[23730]: cluster 2023-07-10T16:59:17.407816+0000 mgr.y (mgr.14144) 162 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:18.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:18 smithi192 bash[23730]: audit 2023-07-10T16:59:17.512290+0000 mon.a (mon.0) 428 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:18.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:18 smithi192 bash[23730]: audit 2023-07-10T16:59:17.515899+0000 mon.a (mon.0) 429 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:59:18.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:18 smithi192 bash[23730]: audit 2023-07-10T16:59:17.524250+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:18.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:18 smithi192 bash[23730]: cluster 2023-07-10T16:59:18.428963+0000 mon.a (mon.0) 431 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2023-07-10T16:59:18.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:18 smithi116 bash[23740]: cephadm 2023-07-10T16:59:17.249631+0000 mgr.y (mgr.14144) 161 : cephadm [INF] Detected new or changed devices on smithi192 2023-07-10T16:59:18.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:18 smithi116 bash[23740]: cluster 2023-07-10T16:59:17.407816+0000 mgr.y (mgr.14144) 162 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:18.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:18 smithi116 bash[23740]: audit 2023-07-10T16:59:17.512290+0000 mon.a (mon.0) 428 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:18.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:18 smithi116 bash[23740]: audit 2023-07-10T16:59:17.515899+0000 mon.a (mon.0) 429 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:59:18.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:18 smithi116 bash[23740]: audit 2023-07-10T16:59:17.524250+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:18.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:18 smithi116 bash[23740]: cluster 2023-07-10T16:59:18.428963+0000 mon.a (mon.0) 431 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2023-07-10T16:59:18.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:18 smithi116 bash[28351]: cephadm 2023-07-10T16:59:17.249631+0000 mgr.y (mgr.14144) 161 : cephadm [INF] Detected new or changed devices on smithi192 2023-07-10T16:59:18.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:18 smithi116 bash[28351]: cluster 2023-07-10T16:59:17.407816+0000 mgr.y (mgr.14144) 162 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:18.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:18 smithi116 bash[28351]: audit 2023-07-10T16:59:17.512290+0000 mon.a (mon.0) 428 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:18.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:18 smithi116 bash[28351]: audit 2023-07-10T16:59:17.515899+0000 mon.a (mon.0) 429 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:59:18.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:18 smithi116 bash[28351]: audit 2023-07-10T16:59:17.524250+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:18.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:18 smithi116 bash[28351]: cluster 2023-07-10T16:59:18.428963+0000 mon.a (mon.0) 431 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2023-07-10T16:59:20.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:20 smithi192 bash[23730]: cluster 2023-07-10T16:59:19.408389+0000 mgr.y (mgr.14144) 163 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:20.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:20 smithi192 bash[23730]: audit 2023-07-10T16:59:19.564383+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T16:59:20.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:20 smithi192 bash[23730]: audit 2023-07-10T16:59:19.567268+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T16:59:20.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:20 smithi192 bash[23730]: audit 2023-07-10T16:59:19.982076+0000 mon.b (mon.2) 17 : audit [INF] from='client.? 172.21.15.192:0/2024520736' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e80e7f2c-5c06-41dd-9e63-5ce0d78b58ce"}]: dispatch 2023-07-10T16:59:20.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:20 smithi192 bash[23730]: audit 2023-07-10T16:59:19.982975+0000 mon.a (mon.0) 434 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e80e7f2c-5c06-41dd-9e63-5ce0d78b58ce"}]: dispatch 2023-07-10T16:59:20.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:20 smithi192 bash[23730]: audit 2023-07-10T16:59:19.992058+0000 mon.a (mon.0) 435 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e80e7f2c-5c06-41dd-9e63-5ce0d78b58ce"}]': finished 2023-07-10T16:59:20.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:20 smithi192 bash[23730]: cluster 2023-07-10T16:59:19.992284+0000 mon.a (mon.0) 436 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2023-07-10T16:59:20.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:20 smithi192 bash[23730]: audit 2023-07-10T16:59:19.992589+0000 mon.a (mon.0) 437 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T16:59:20.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:20 smithi116 bash[23740]: cluster 2023-07-10T16:59:19.408389+0000 mgr.y (mgr.14144) 163 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:20.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:20 smithi116 bash[23740]: audit 2023-07-10T16:59:19.564383+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T16:59:20.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:20 smithi116 bash[23740]: audit 2023-07-10T16:59:19.567268+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T16:59:20.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:20 smithi116 bash[23740]: audit 2023-07-10T16:59:19.982076+0000 mon.b (mon.2) 17 : audit [INF] from='client.? 172.21.15.192:0/2024520736' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e80e7f2c-5c06-41dd-9e63-5ce0d78b58ce"}]: dispatch 2023-07-10T16:59:20.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:20 smithi116 bash[23740]: audit 2023-07-10T16:59:19.982975+0000 mon.a (mon.0) 434 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e80e7f2c-5c06-41dd-9e63-5ce0d78b58ce"}]: dispatch 2023-07-10T16:59:20.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:20 smithi116 bash[23740]: audit 2023-07-10T16:59:19.992058+0000 mon.a (mon.0) 435 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e80e7f2c-5c06-41dd-9e63-5ce0d78b58ce"}]': finished 2023-07-10T16:59:20.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:20 smithi116 bash[23740]: cluster 2023-07-10T16:59:19.992284+0000 mon.a (mon.0) 436 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2023-07-10T16:59:20.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:20 smithi116 bash[23740]: audit 2023-07-10T16:59:19.992589+0000 mon.a (mon.0) 437 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T16:59:20.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:20 smithi116 bash[28351]: cluster 2023-07-10T16:59:19.408389+0000 mgr.y (mgr.14144) 163 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:20.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:20 smithi116 bash[28351]: audit 2023-07-10T16:59:19.564383+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T16:59:20.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:20 smithi116 bash[28351]: audit 2023-07-10T16:59:19.567268+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T16:59:20.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:20 smithi116 bash[28351]: audit 2023-07-10T16:59:19.982076+0000 mon.b (mon.2) 17 : audit [INF] from='client.? 172.21.15.192:0/2024520736' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e80e7f2c-5c06-41dd-9e63-5ce0d78b58ce"}]: dispatch 2023-07-10T16:59:20.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:20 smithi116 bash[28351]: audit 2023-07-10T16:59:19.982975+0000 mon.a (mon.0) 434 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e80e7f2c-5c06-41dd-9e63-5ce0d78b58ce"}]: dispatch 2023-07-10T16:59:20.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:20 smithi116 bash[28351]: audit 2023-07-10T16:59:19.992058+0000 mon.a (mon.0) 435 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e80e7f2c-5c06-41dd-9e63-5ce0d78b58ce"}]': finished 2023-07-10T16:59:20.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:20 smithi116 bash[28351]: cluster 2023-07-10T16:59:19.992284+0000 mon.a (mon.0) 436 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2023-07-10T16:59:20.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:20 smithi116 bash[28351]: audit 2023-07-10T16:59:19.992589+0000 mon.a (mon.0) 437 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T16:59:21.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:21 smithi192 bash[23730]: audit 2023-07-10T16:59:20.645125+0000 mon.b (mon.2) 18 : audit [DBG] from='client.? 172.21.15.192:0/1291133123' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:59:21.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:21 smithi116 bash[28351]: audit 2023-07-10T16:59:20.645125+0000 mon.b (mon.2) 18 : audit [DBG] from='client.? 172.21.15.192:0/1291133123' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:59:21.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:21 smithi116 bash[23740]: audit 2023-07-10T16:59:20.645125+0000 mon.b (mon.2) 18 : audit [DBG] from='client.? 172.21.15.192:0/1291133123' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-10T16:59:22.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:22 smithi192 bash[23730]: cluster 2023-07-10T16:59:21.409048+0000 mgr.y (mgr.14144) 164 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:22.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:22 smithi116 bash[23740]: cluster 2023-07-10T16:59:21.409048+0000 mgr.y (mgr.14144) 164 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:22.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:22 smithi116 bash[28351]: cluster 2023-07-10T16:59:21.409048+0000 mgr.y (mgr.14144) 164 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:24.731 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:24 smithi192 bash[23730]: cluster 2023-07-10T16:59:23.409637+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:24.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:24 smithi116 bash[23740]: cluster 2023-07-10T16:59:23.409637+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:24.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:24 smithi116 bash[28351]: cluster 2023-07-10T16:59:23.409637+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:26.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:26 smithi192 bash[23730]: cluster 2023-07-10T16:59:25.410319+0000 mgr.y (mgr.14144) 166 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:26.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:26 smithi116 bash[23740]: cluster 2023-07-10T16:59:25.410319+0000 mgr.y (mgr.14144) 166 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:26.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:26 smithi116 bash[28351]: cluster 2023-07-10T16:59:25.410319+0000 mgr.y (mgr.14144) 166 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:28.699 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:28 smithi192 bash[23730]: cluster 2023-07-10T16:59:27.410913+0000 mgr.y (mgr.14144) 167 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:28.700 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:28 smithi192 bash[23730]: audit 2023-07-10T16:59:27.599554+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-10T16:59:28.700 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:28 smithi192 bash[23730]: audit 2023-07-10T16:59:27.601362+0000 mon.a (mon.0) 439 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:59:28.700 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:28 smithi192 bash[23730]: cephadm 2023-07-10T16:59:27.602674+0000 mgr.y (mgr.14144) 168 : cephadm [INF] Deploying daemon osd.7 on smithi192 2023-07-10T16:59:28.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:28 smithi116 bash[23740]: cluster 2023-07-10T16:59:27.410913+0000 mgr.y (mgr.14144) 167 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:28.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:28 smithi116 bash[23740]: audit 2023-07-10T16:59:27.599554+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-10T16:59:28.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:28 smithi116 bash[23740]: audit 2023-07-10T16:59:27.601362+0000 mon.a (mon.0) 439 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:59:28.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:28 smithi116 bash[23740]: cephadm 2023-07-10T16:59:27.602674+0000 mgr.y (mgr.14144) 168 : cephadm [INF] Deploying daemon osd.7 on smithi192 2023-07-10T16:59:28.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:28 smithi116 bash[28351]: cluster 2023-07-10T16:59:27.410913+0000 mgr.y (mgr.14144) 167 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:28.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:28 smithi116 bash[28351]: audit 2023-07-10T16:59:27.599554+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-10T16:59:28.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:28 smithi116 bash[28351]: audit 2023-07-10T16:59:27.601362+0000 mon.a (mon.0) 439 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T16:59:28.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:28 smithi116 bash[28351]: cephadm 2023-07-10T16:59:27.602674+0000 mgr.y (mgr.14144) 168 : cephadm [INF] Deploying daemon osd.7 on smithi192 2023-07-10T16:59:30.017 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 7 on host 'smithi192' 2023-07-10T16:59:30.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:30 smithi192 bash[23730]: cluster 2023-07-10T16:59:29.411574+0000 mgr.y (mgr.14144) 169 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:30.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:30 smithi192 bash[23730]: audit 2023-07-10T16:59:29.995707+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:59:30.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:30 smithi192 bash[23730]: audit 2023-07-10T16:59:30.005482+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:30.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:30 smithi192 bash[23730]: audit 2023-07-10T16:59:30.013363+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:30.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:30 smithi116 bash[23740]: cluster 2023-07-10T16:59:29.411574+0000 mgr.y (mgr.14144) 169 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:30.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:30 smithi116 bash[23740]: audit 2023-07-10T16:59:29.995707+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:59:30.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:30 smithi116 bash[23740]: audit 2023-07-10T16:59:30.005482+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:30.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:30 smithi116 bash[23740]: audit 2023-07-10T16:59:30.013363+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:30.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:30 smithi116 bash[28351]: cluster 2023-07-10T16:59:29.411574+0000 mgr.y (mgr.14144) 169 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:30.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:30 smithi116 bash[28351]: audit 2023-07-10T16:59:29.995707+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:59:30.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:30 smithi116 bash[28351]: audit 2023-07-10T16:59:30.005482+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:30.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:30 smithi116 bash[28351]: audit 2023-07-10T16:59:30.013363+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:31.175 DEBUG:teuthology.orchestra.run.smithi192:osd.7> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.7.service 2023-07-10T16:59:31.178 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2023-07-10T16:59:31.178 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-10T16:59:31.393 INFO:journalctl@ceph.osd.7.smithi192.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T16:59:32.136 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 16:59:31 smithi192 bash[34677]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-07-10T16:59:32.137 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 16:59:31 smithi192 bash[34677]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-7 --no-mon-config 2023-07-10T16:59:32.137 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 16:59:31 smithi192 bash[34677]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-7/block 2023-07-10T16:59:32.137 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 16:59:31 smithi192 bash[34677]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-7/block 2023-07-10T16:59:32.138 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 16:59:31 smithi192 bash[34677]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2023-07-10T16:59:32.138 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 16:59:31 smithi192 bash[34677]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-07-10T16:59:32.138 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 16:59:31 smithi192 bash[34677]: --> ceph-volume lvm activate successful for osd ID: 7 2023-07-10T16:59:32.595 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:32 smithi116 bash[23740]: cluster 2023-07-10T16:59:31.412125+0000 mgr.y (mgr.14144) 170 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:32.596 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:32 smithi116 bash[28351]: cluster 2023-07-10T16:59:31.412125+0000 mgr.y (mgr.14144) 170 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:32.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:32 smithi192 bash[23730]: cluster 2023-07-10T16:59:31.412125+0000 mgr.y (mgr.14144) 170 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:33.630 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 16:59:33 smithi192 bash[34832]: debug 2023-07-10T16:59:33.343+0000 7f2d48bce700 1 -- 172.21.15.192:0/2890298246 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dcac2a44e0 con 0x55dcac31cc00 2023-07-10T16:59:34.307 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-07-10T16:59:34.386 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 16:59:33 smithi192 bash[34832]: debug 2023-07-10T16:59:33.895+0000 7f2d50659080 -1 Falling back to public interface 2023-07-10T16:59:34.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:34 smithi116 bash[28351]: cluster 2023-07-10T16:59:33.412785+0000 mgr.y (mgr.14144) 171 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:34.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:34 smithi116 bash[28351]: audit 2023-07-10T16:59:33.438337+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:34.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:34 smithi116 bash[28351]: audit 2023-07-10T16:59:34.304735+0000 mon.a (mon.0) 444 : audit [DBG] from='client.? 172.21.15.116:0/2845753930' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-10T16:59:34.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:34 smithi116 bash[23740]: cluster 2023-07-10T16:59:33.412785+0000 mgr.y (mgr.14144) 171 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:34.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:34 smithi116 bash[23740]: audit 2023-07-10T16:59:33.438337+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:34.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:34 smithi116 bash[23740]: audit 2023-07-10T16:59:34.304735+0000 mon.a (mon.0) 444 : audit [DBG] from='client.? 172.21.15.116:0/2845753930' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-10T16:59:34.726 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:34 smithi192 bash[23730]: cluster 2023-07-10T16:59:33.412785+0000 mgr.y (mgr.14144) 171 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:34.727 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:34 smithi192 bash[23730]: audit 2023-07-10T16:59:33.438337+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:34.727 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:34 smithi192 bash[23730]: audit 2023-07-10T16:59:34.304735+0000 mon.a (mon.0) 444 : audit [DBG] from='client.? 172.21.15.116:0/2845753930' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-10T16:59:34.970 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":46,"num_osds":8,"num_up_osds":7,"osd_up_since":1689008356,"num_in_osds":8,"osd_in_since":1689008359,"num_remapped_pgs":0} 2023-07-10T16:59:35.972 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-10T16:59:36.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:36 smithi192 bash[23730]: cluster 2023-07-10T16:59:35.413396+0000 mgr.y (mgr.14144) 172 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:36.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:36 smithi116 bash[23740]: cluster 2023-07-10T16:59:35.413396+0000 mgr.y (mgr.14144) 172 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:36.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:36 smithi116 bash[28351]: cluster 2023-07-10T16:59:35.413396+0000 mgr.y (mgr.14144) 172 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:37.386 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 16:59:36 smithi192 bash[34832]: debug 2023-07-10T16:59:36.939+0000 7f2d50659080 -1 osd.7 0 log_to_monitors {default=true} 2023-07-10T16:59:37.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:37 smithi192 bash[23730]: audit 2023-07-10T16:59:36.948695+0000 mon.b (mon.2) 19 : audit [INF] from='osd.7 [v2:172.21.15.192:6824/2832282377,v1:172.21.15.192:6825/2832282377]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-10T16:59:37.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:37 smithi192 bash[23730]: audit 2023-07-10T16:59:36.949708+0000 mon.a (mon.0) 445 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-10T16:59:37.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:37 smithi116 bash[23740]: audit 2023-07-10T16:59:36.948695+0000 mon.b (mon.2) 19 : audit [INF] from='osd.7 [v2:172.21.15.192:6824/2832282377,v1:172.21.15.192:6825/2832282377]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-10T16:59:37.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:37 smithi116 bash[23740]: audit 2023-07-10T16:59:36.949708+0000 mon.a (mon.0) 445 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-10T16:59:37.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:37 smithi116 bash[28351]: audit 2023-07-10T16:59:36.948695+0000 mon.b (mon.2) 19 : audit [INF] from='osd.7 [v2:172.21.15.192:6824/2832282377,v1:172.21.15.192:6825/2832282377]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-10T16:59:37.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:37 smithi116 bash[28351]: audit 2023-07-10T16:59:36.949708+0000 mon.a (mon.0) 445 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-10T16:59:38.886 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 16:59:38 smithi192 bash[34832]: debug 2023-07-10T16:59:38.467+0000 7f2d46bca700 -1 osd.7 0 waiting for initial osdmap 2023-07-10T16:59:38.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:38 smithi192 bash[23730]: cluster 2023-07-10T16:59:37.413966+0000 mgr.y (mgr.14144) 173 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:38.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:38 smithi192 bash[23730]: audit 2023-07-10T16:59:37.454659+0000 mon.a (mon.0) 446 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-10T16:59:38.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:38 smithi192 bash[23730]: cluster 2023-07-10T16:59:37.454801+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2023-07-10T16:59:38.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:38 smithi192 bash[23730]: audit 2023-07-10T16:59:37.455056+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T16:59:38.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:38 smithi192 bash[23730]: audit 2023-07-10T16:59:37.456580+0000 mon.b (mon.2) 20 : audit [INF] from='osd.7 [v2:172.21.15.192:6824/2832282377,v1:172.21.15.192:6825/2832282377]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:59:38.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:38 smithi192 bash[23730]: audit 2023-07-10T16:59:37.457381+0000 mon.a (mon.0) 449 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:59:38.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:38 smithi192 bash[23730]: audit 2023-07-10T16:59:38.291548+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:38.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:38 smithi192 bash[23730]: audit 2023-07-10T16:59:38.299231+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:38.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:38 smithi192 bash[23730]: audit 2023-07-10T16:59:38.301484+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:59:38.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:38 smithi192 bash[23730]: audit 2023-07-10T16:59:38.309059+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:38.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[23740]: cluster 2023-07-10T16:59:37.413966+0000 mgr.y (mgr.14144) 173 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:38.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[23740]: audit 2023-07-10T16:59:37.454659+0000 mon.a (mon.0) 446 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-10T16:59:38.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[23740]: cluster 2023-07-10T16:59:37.454801+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2023-07-10T16:59:38.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[23740]: audit 2023-07-10T16:59:37.455056+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T16:59:38.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[23740]: audit 2023-07-10T16:59:37.456580+0000 mon.b (mon.2) 20 : audit [INF] from='osd.7 [v2:172.21.15.192:6824/2832282377,v1:172.21.15.192:6825/2832282377]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:59:38.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[23740]: audit 2023-07-10T16:59:37.457381+0000 mon.a (mon.0) 449 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:59:38.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[23740]: audit 2023-07-10T16:59:38.291548+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:38.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[23740]: audit 2023-07-10T16:59:38.299231+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:38.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[23740]: audit 2023-07-10T16:59:38.301484+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:59:38.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[23740]: audit 2023-07-10T16:59:38.309059+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:38.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[28351]: cluster 2023-07-10T16:59:37.413966+0000 mgr.y (mgr.14144) 173 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:38.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[28351]: audit 2023-07-10T16:59:37.454659+0000 mon.a (mon.0) 446 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-10T16:59:38.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[28351]: cluster 2023-07-10T16:59:37.454801+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2023-07-10T16:59:38.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[28351]: audit 2023-07-10T16:59:37.455056+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T16:59:38.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[28351]: audit 2023-07-10T16:59:37.456580+0000 mon.b (mon.2) 20 : audit [INF] from='osd.7 [v2:172.21.15.192:6824/2832282377,v1:172.21.15.192:6825/2832282377]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:59:38.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[28351]: audit 2023-07-10T16:59:37.457381+0000 mon.a (mon.0) 449 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T16:59:38.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[28351]: audit 2023-07-10T16:59:38.291548+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:38.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[28351]: audit 2023-07-10T16:59:38.299231+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:38.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[28351]: audit 2023-07-10T16:59:38.301484+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T16:59:38.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:38 smithi116 bash[28351]: audit 2023-07-10T16:59:38.309059+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T16:59:39.316 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-07-10T16:59:39.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:39 smithi116 bash[23740]: cephadm 2023-07-10T16:59:38.279433+0000 mgr.y (mgr.14144) 174 : cephadm [INF] Detected new or changed devices on smithi192 2023-07-10T16:59:39.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:39 smithi116 bash[23740]: audit 2023-07-10T16:59:38.459832+0000 mon.a (mon.0) 454 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-10T16:59:39.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:39 smithi116 bash[23740]: cluster 2023-07-10T16:59:38.460194+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e48: 8 total, 7 up, 8 in 2023-07-10T16:59:39.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:39 smithi116 bash[23740]: audit 2023-07-10T16:59:38.461084+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T16:59:39.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:39 smithi116 bash[23740]: audit 2023-07-10T16:59:38.468626+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T16:59:39.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:39 smithi116 bash[23740]: audit 2023-07-10T16:59:39.313832+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.116:0/3608671518' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-10T16:59:39.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:39 smithi116 bash[28351]: cephadm 2023-07-10T16:59:38.279433+0000 mgr.y (mgr.14144) 174 : cephadm [INF] Detected new or changed devices on smithi192 2023-07-10T16:59:39.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:39 smithi116 bash[28351]: audit 2023-07-10T16:59:38.459832+0000 mon.a (mon.0) 454 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-10T16:59:39.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:39 smithi116 bash[28351]: cluster 2023-07-10T16:59:38.460194+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e48: 8 total, 7 up, 8 in 2023-07-10T16:59:39.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:39 smithi116 bash[28351]: audit 2023-07-10T16:59:38.461084+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T16:59:39.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:39 smithi116 bash[28351]: audit 2023-07-10T16:59:38.468626+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T16:59:39.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:39 smithi116 bash[28351]: audit 2023-07-10T16:59:39.313832+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.116:0/3608671518' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-10T16:59:39.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:39 smithi192 bash[23730]: cephadm 2023-07-10T16:59:38.279433+0000 mgr.y (mgr.14144) 174 : cephadm [INF] Detected new or changed devices on smithi192 2023-07-10T16:59:39.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:39 smithi192 bash[23730]: audit 2023-07-10T16:59:38.459832+0000 mon.a (mon.0) 454 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-07-10T16:59:39.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:39 smithi192 bash[23730]: cluster 2023-07-10T16:59:38.460194+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e48: 8 total, 7 up, 8 in 2023-07-10T16:59:39.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:39 smithi192 bash[23730]: audit 2023-07-10T16:59:38.461084+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T16:59:39.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:39 smithi192 bash[23730]: audit 2023-07-10T16:59:38.468626+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T16:59:39.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:39 smithi192 bash[23730]: audit 2023-07-10T16:59:39.313832+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.116:0/3608671518' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-10T16:59:40.078 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":48,"num_osds":8,"num_up_osds":7,"osd_up_since":1689008356,"num_in_osds":8,"osd_in_since":1689008359,"num_remapped_pgs":0} 2023-07-10T16:59:40.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:40 smithi192 bash[23730]: cluster 2023-07-10T16:59:37.987393+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:59:40.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:40 smithi192 bash[23730]: cluster 2023-07-10T16:59:37.987596+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:59:40.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:40 smithi192 bash[23730]: cluster 2023-07-10T16:59:39.414524+0000 mgr.y (mgr.14144) 175 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:40.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:40 smithi192 bash[23730]: cluster 2023-07-10T16:59:39.463288+0000 mon.a (mon.0) 458 : cluster [INF] osd.7 [v2:172.21.15.192:6824/2832282377,v1:172.21.15.192:6825/2832282377] boot 2023-07-10T16:59:40.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:40 smithi192 bash[23730]: cluster 2023-07-10T16:59:39.463378+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2023-07-10T16:59:40.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:40 smithi192 bash[23730]: audit 2023-07-10T16:59:39.463860+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T16:59:40.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:40 smithi116 bash[23740]: cluster 2023-07-10T16:59:37.987393+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:59:40.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:40 smithi116 bash[23740]: cluster 2023-07-10T16:59:37.987596+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:59:40.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:40 smithi116 bash[23740]: cluster 2023-07-10T16:59:39.414524+0000 mgr.y (mgr.14144) 175 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:40.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:40 smithi116 bash[23740]: cluster 2023-07-10T16:59:39.463288+0000 mon.a (mon.0) 458 : cluster [INF] osd.7 [v2:172.21.15.192:6824/2832282377,v1:172.21.15.192:6825/2832282377] boot 2023-07-10T16:59:40.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:40 smithi116 bash[23740]: cluster 2023-07-10T16:59:39.463378+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2023-07-10T16:59:40.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:40 smithi116 bash[23740]: audit 2023-07-10T16:59:39.463860+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T16:59:40.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:40 smithi116 bash[28351]: cluster 2023-07-10T16:59:37.987393+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2023-07-10T16:59:40.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:40 smithi116 bash[28351]: cluster 2023-07-10T16:59:37.987596+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2023-07-10T16:59:40.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:40 smithi116 bash[28351]: cluster 2023-07-10T16:59:39.414524+0000 mgr.y (mgr.14144) 175 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2023-07-10T16:59:40.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:40 smithi116 bash[28351]: cluster 2023-07-10T16:59:39.463288+0000 mon.a (mon.0) 458 : cluster [INF] osd.7 [v2:172.21.15.192:6824/2832282377,v1:172.21.15.192:6825/2832282377] boot 2023-07-10T16:59:40.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:40 smithi116 bash[28351]: cluster 2023-07-10T16:59:39.463378+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2023-07-10T16:59:40.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:40 smithi116 bash[28351]: audit 2023-07-10T16:59:39.463860+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T16:59:41.079 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-10T16:59:41.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:41 smithi116 bash[23740]: cluster 2023-07-10T16:59:40.467877+0000 mon.a (mon.0) 461 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2023-07-10T16:59:41.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:41 smithi116 bash[23740]: cluster 2023-07-10T16:59:41.466143+0000 mon.a (mon.0) 462 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2023-07-10T16:59:41.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:41 smithi116 bash[28351]: cluster 2023-07-10T16:59:40.467877+0000 mon.a (mon.0) 461 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2023-07-10T16:59:41.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:41 smithi116 bash[28351]: cluster 2023-07-10T16:59:41.466143+0000 mon.a (mon.0) 462 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2023-07-10T16:59:41.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:41 smithi192 bash[23730]: cluster 2023-07-10T16:59:40.467877+0000 mon.a (mon.0) 461 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2023-07-10T16:59:41.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:41 smithi192 bash[23730]: cluster 2023-07-10T16:59:41.466143+0000 mon.a (mon.0) 462 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2023-07-10T16:59:42.544 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:42 smithi116 bash[28351]: cluster 2023-07-10T16:59:41.415147+0000 mgr.y (mgr.14144) 176 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:42.544 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:42 smithi116 bash[23740]: cluster 2023-07-10T16:59:41.415147+0000 mgr.y (mgr.14144) 176 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:42.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:42 smithi192 bash[23730]: cluster 2023-07-10T16:59:41.415147+0000 mgr.y (mgr.14144) 176 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:44.287 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-07-10T16:59:44.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:44 smithi116 bash[23740]: cluster 2023-07-10T16:59:43.415876+0000 mgr.y (mgr.14144) 177 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:44.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:44 smithi116 bash[23740]: audit 2023-07-10T16:59:44.284985+0000 mon.a (mon.0) 463 : audit [DBG] from='client.? 172.21.15.116:0/646389364' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-10T16:59:44.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:44 smithi116 bash[28351]: cluster 2023-07-10T16:59:43.415876+0000 mgr.y (mgr.14144) 177 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:44.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:44 smithi116 bash[28351]: audit 2023-07-10T16:59:44.284985+0000 mon.a (mon.0) 463 : audit [DBG] from='client.? 172.21.15.116:0/646389364' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-10T16:59:44.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:44 smithi192 bash[23730]: cluster 2023-07-10T16:59:43.415876+0000 mgr.y (mgr.14144) 177 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:44.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:44 smithi192 bash[23730]: audit 2023-07-10T16:59:44.284985+0000 mon.a (mon.0) 463 : audit [DBG] from='client.? 172.21.15.116:0/646389364' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-10T16:59:45.026 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":51,"num_osds":8,"num_up_osds":8,"osd_up_since":1689008379,"num_in_osds":8,"osd_in_since":1689008359,"num_remapped_pgs":0} 2023-07-10T16:59:45.027 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph osd dump --format=json 2023-07-10T16:59:46.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:46 smithi192 bash[23730]: cluster 2023-07-10T16:59:45.416554+0000 mgr.y (mgr.14144) 178 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:46.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:46 smithi116 bash[28351]: cluster 2023-07-10T16:59:45.416554+0000 mgr.y (mgr.14144) 178 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:46.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:46 smithi116 bash[23740]: cluster 2023-07-10T16:59:45.416554+0000 mgr.y (mgr.14144) 178 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:47.769 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T16:59:48.740 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:48 smithi116 bash[23740]: cluster 2023-07-10T16:59:47.417188+0000 mgr.y (mgr.14144) 179 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:48.741 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:48 smithi116 bash[28351]: cluster 2023-07-10T16:59:47.417188+0000 mgr.y (mgr.14144) 179 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:48.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:48 smithi192 bash[23730]: cluster 2023-07-10T16:59:47.417188+0000 mgr.y (mgr.14144) 179 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:50.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:50 smithi192 bash[23730]: cluster 2023-07-10T16:59:49.417833+0000 mgr.y (mgr.14144) 180 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:50.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:50 smithi116 bash[23740]: cluster 2023-07-10T16:59:49.417833+0000 mgr.y (mgr.14144) 180 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:50.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:50 smithi116 bash[28351]: cluster 2023-07-10T16:59:49.417833+0000 mgr.y (mgr.14144) 180 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:50.953 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-07-10T16:59:50.953 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":51,"fsid":"3f83c2ec-1f42-11ee-9b31-001a4aab830c","created":"2023-07-10T16:54:21.709029+0000","modified":"2023-07-10T16:59:41.458155+0000","last_up_change":"2023-07-10T16:59:39.452978+0000","last_in_change":"2023-07-10T16:59:19.984246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-07-10T16:57:56.390654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"9234f7ee-8413-46af-98b7-6b957be6dd6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2297482263},{"type":"v1","addr":"172.21.15.116:6803","nonce":2297482263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2297482263},{"type":"v1","addr":"172.21.15.116:6805","nonce":2297482263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2297482263},{"type":"v1","addr":"172.21.15.116:6809","nonce":2297482263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2297482263},{"type":"v1","addr":"172.21.15.116:6807","nonce":2297482263}]},"public_addr":"172.21.15.116:6803/2297482263","cluster_addr":"172.21.15.116:6805/2297482263","heartbeat_back_addr":"172.21.15.116:6809/2297482263","heartbeat_front_addr":"172.21.15.116:6807/2297482263","state":["exists","up"]},{"osd":1,"uuid":"6d22b106-7d49-4c50-8c01-c018ae237563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3104401754},{"type":"v1","addr":"172.21.15.116:6811","nonce":3104401754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3104401754},{"type":"v1","addr":"172.21.15.116:6813","nonce":3104401754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":3104401754},{"type":"v1","addr":"172.21.15.116:6817","nonce":3104401754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3104401754},{"type":"v1","addr":"172.21.15.116:6815","nonce":3104401754}]},"public_addr":"172.21.15.116:6811/3104401754","cluster_addr":"172.21.15.116:6813/3104401754","heartbeat_back_addr":"172.21.15.116:6817/3104401754","heartbeat_front_addr":"172.21.15.116:6815/3104401754","state":["exists","up"]},{"osd":2,"uuid":"f1368d93-6cb3-4fe8-b945-974ee8982326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1907366872},{"type":"v1","addr":"172.21.15.116:6819","nonce":1907366872}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1907366872},{"type":"v1","addr":"172.21.15.116:6821","nonce":1907366872}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1907366872},{"type":"v1","addr":"172.21.15.116:6825","nonce":1907366872}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1907366872},{"type":"v1","addr":"172.21.15.116:6823","nonce":1907366872}]},"public_addr":"172.21.15.116:6819/1907366872","cluster_addr":"172.21.15.116:6821/1907366872","heartbeat_back_addr":"172.21.15.116:6825/1907366872","heartbeat_front_addr":"172.21.15.116:6823/1907366872","state":["exists","up"]},{"osd":3,"uuid":"d02ea7dd-deab-4a14-b7d2-56e45f1e51e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":278136235},{"type":"v1","addr":"172.21.15.116:6827","nonce":278136235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":278136235},{"type":"v1","addr":"172.21.15.116:6829","nonce":278136235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":278136235},{"type":"v1","addr":"172.21.15.116:6833","nonce":278136235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":278136235},{"type":"v1","addr":"172.21.15.116:6831","nonce":278136235}]},"public_addr":"172.21.15.116:6827/278136235","cluster_addr":"172.21.15.116:6829/278136235","heartbeat_back_addr":"172.21.15.116:6833/278136235","heartbeat_front_addr":"172.21.15.116:6831/278136235","state":["exists","up"]},{"osd":4,"uuid":"de3dac78-358c-4e5d-ae57-445a54a5c29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":1383435807},{"type":"v1","addr":"172.21.15.192:6801","nonce":1383435807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":1383435807},{"type":"v1","addr":"172.21.15.192:6803","nonce":1383435807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":1383435807},{"type":"v1","addr":"172.21.15.192:6807","nonce":1383435807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":1383435807},{"type":"v1","addr":"172.21.15.192:6805","nonce":1383435807}]},"public_addr":"172.21.15.192:6801/1383435807","cluster_addr":"172.21.15.192:6803/1383435807","heartbeat_back_addr":"172.21.15.192:6807/1383435807","heartbeat_front_addr":"172.21.15.192:6805/1383435807","state":["exists","up"]},{"osd":5,"uuid":"14218aa2-8ec8-407f-879b-b4fc762177fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2450645600},{"type":"v1","addr":"172.21.15.192:6809","nonce":2450645600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2450645600},{"type":"v1","addr":"172.21.15.192:6811","nonce":2450645600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2450645600},{"type":"v1","addr":"172.21.15.192:6815","nonce":2450645600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2450645600},{"type":"v1","addr":"172.21.15.192:6813","nonce":2450645600}]},"public_addr":"172.21.15.192:6809/2450645600","cluster_addr":"172.21.15.192:6811/2450645600","heartbeat_back_addr":"172.21.15.192:6815/2450645600","heartbeat_front_addr":"172.21.15.192:6813/2450645600","state":["exists","up"]},{"osd":6,"uuid":"b279a6e4-a920-46c2-b19b-59131f5c68f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":425451514},{"type":"v1","addr":"172.21.15.192:6817","nonce":425451514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":425451514},{"type":"v1","addr":"172.21.15.192:6819","nonce":425451514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":425451514},{"type":"v1","addr":"172.21.15.192:6823","nonce":425451514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":425451514},{"type":"v1","addr":"172.21.15.192:6821","nonce":425451514}]},"public_addr":"172.21.15.192:6817/425451514","cluster_addr":"172.21.15.192:6819/425451514","heartbeat_back_addr":"172.21.15.192:6823/425451514","heartbeat_front_addr":"172.21.15.192:6821/425451514","state":["exists","up"]},{"osd":7,"uuid":"e80e7f2c-5c06-41dd-9e63-5ce0d78b58ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2832282377},{"type":"v1","addr":"172.21.15.192:6825","nonce":2832282377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2832282377},{"type":"v1","addr":"172.21.15.192:6827","nonce":2832282377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2832282377},{"type":"v1","addr":"172.21.15.192:6831","nonce":2832282377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2832282377},{"type":"v1","addr":"172.21.15.192:6829","nonce":2832282377}]},"public_addr":"172.21.15.192:6825/2832282377","cluster_addr":"172.21.15.192:6827/2832282377","heartbeat_back_addr":"172.21.15.192:6831/2832282377","heartbeat_front_addr":"172.21.15.192:6829/2832282377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:56:45.784382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:57:11.613844+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:57:39.816175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:58:03.754473+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:58:25.910824+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:58:50.215523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:59:14.648618+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:59:37.987601+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/1249272063":"2023-07-11T16:55:18.481033+0000","172.21.15.116:0/3507856810":"2023-07-11T16:55:18.481033+0000","172.21.15.116:0/221490838":"2023-07-11T16:55:18.481033+0000","172.21.15.116:6801/1249272063":"2023-07-11T16:55:18.481033+0000","172.21.15.116:0/234488842":"2023-07-11T16:54:49.149280+0000","172.21.15.116:6800/48925241":"2023-07-11T16:54:49.149280+0000","172.21.15.116:6801/48925241":"2023-07-11T16:54:49.149280+0000","172.21.15.116:0/1382514600":"2023-07-11T16:54:49.149280+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-10T16:59:51.670 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-07-10T16:57:56.390654+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2023-07-10T16:59:51.671 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph osd pool get device_health_metrics pg_num 2023-07-10T16:59:51.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:51 smithi192 bash[23730]: audit 2023-07-10T16:59:50.948976+0000 mon.a (mon.0) 464 : audit [DBG] from='client.? 172.21.15.116:0/1957712321' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-10T16:59:51.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:51 smithi116 bash[23740]: audit 2023-07-10T16:59:50.948976+0000 mon.a (mon.0) 464 : audit [DBG] from='client.? 172.21.15.116:0/1957712321' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-10T16:59:51.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:51 smithi116 bash[28351]: audit 2023-07-10T16:59:50.948976+0000 mon.a (mon.0) 464 : audit [DBG] from='client.? 172.21.15.116:0/1957712321' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-10T16:59:52.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:52 smithi192 bash[23730]: cluster 2023-07-10T16:59:51.418483+0000 mgr.y (mgr.14144) 181 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:52.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:52 smithi116 bash[23740]: cluster 2023-07-10T16:59:51.418483+0000 mgr.y (mgr.14144) 181 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:52.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:52 smithi116 bash[28351]: cluster 2023-07-10T16:59:51.418483+0000 mgr.y (mgr.14144) 181 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:54.397 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T16:59:54.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:54 smithi192 bash[23730]: cluster 2023-07-10T16:59:53.419194+0000 mgr.y (mgr.14144) 182 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:54.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:54 smithi116 bash[23740]: cluster 2023-07-10T16:59:53.419194+0000 mgr.y (mgr.14144) 182 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:54.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:54 smithi116 bash[28351]: cluster 2023-07-10T16:59:53.419194+0000 mgr.y (mgr.14144) 182 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:56.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:56 smithi116 bash[23740]: cluster 2023-07-10T16:59:55.419743+0000 mgr.y (mgr.14144) 183 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:56.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:56 smithi116 bash[28351]: cluster 2023-07-10T16:59:55.419743+0000 mgr.y (mgr.14144) 183 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:56.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:56 smithi192 bash[23730]: cluster 2023-07-10T16:59:55.419743+0000 mgr.y (mgr.14144) 183 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:57.554 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-07-10T16:59:58.248 INFO:tasks.cephadm:Adding prometheus.a on smithi192 2023-07-10T16:59:58.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph orch apply prometheus '1;smithi192=a' 2023-07-10T16:59:58.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:58 smithi192 bash[23730]: cluster 2023-07-10T16:59:57.420301+0000 mgr.y (mgr.14144) 184 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:58.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 16:59:58 smithi192 bash[23730]: audit 2023-07-10T16:59:57.551063+0000 mon.a (mon.0) 465 : audit [DBG] from='client.? 172.21.15.116:0/3249065765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2023-07-10T16:59:58.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:58 smithi116 bash[23740]: cluster 2023-07-10T16:59:57.420301+0000 mgr.y (mgr.14144) 184 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:58.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 16:59:58 smithi116 bash[23740]: audit 2023-07-10T16:59:57.551063+0000 mon.a (mon.0) 465 : audit [DBG] from='client.? 172.21.15.116:0/3249065765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2023-07-10T16:59:58.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:58 smithi116 bash[28351]: cluster 2023-07-10T16:59:57.420301+0000 mgr.y (mgr.14144) 184 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T16:59:58.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 16:59:58 smithi116 bash[28351]: audit 2023-07-10T16:59:57.551063+0000 mon.a (mon.0) 465 : audit [DBG] from='client.? 172.21.15.116:0/3249065765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2023-07-10T17:00:00.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:00 smithi192 bash[23730]: cluster 2023-07-10T16:59:59.420913+0000 mgr.y (mgr.14144) 185 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:00.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:00 smithi192 bash[23730]: cluster 2023-07-10T17:00:00.000159+0000 mon.a (mon.0) 466 : cluster [INF] overall HEALTH_OK 2023-07-10T17:00:00.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:00 smithi116 bash[28351]: cluster 2023-07-10T16:59:59.420913+0000 mgr.y (mgr.14144) 185 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:00.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:00 smithi116 bash[28351]: cluster 2023-07-10T17:00:00.000159+0000 mon.a (mon.0) 466 : cluster [INF] overall HEALTH_OK 2023-07-10T17:00:00.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:00 smithi116 bash[23740]: cluster 2023-07-10T16:59:59.420913+0000 mgr.y (mgr.14144) 185 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:00.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:00 smithi116 bash[23740]: cluster 2023-07-10T17:00:00.000159+0000 mon.a (mon.0) 466 : cluster [INF] overall HEALTH_OK 2023-07-10T17:00:01.302 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled prometheus update... 2023-07-10T17:00:02.010 DEBUG:teuthology.orchestra.run.smithi192:prometheus.a> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service 2023-07-10T17:00:02.013 INFO:tasks.cephadm:Adding node-exporter.a on smithi116 2023-07-10T17:00:02.014 INFO:tasks.cephadm:Adding node-exporter.b on smithi192 2023-07-10T17:00:02.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph orch apply node-exporter '2;smithi116=a;smithi192=b' 2023-07-10T17:00:02.213 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T17:00:02.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:02 smithi192 bash[23730]: audit 2023-07-10T17:00:01.290405+0000 mgr.y (mgr.14144) 186 : audit [DBG] from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:00:02.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:02 smithi192 bash[23730]: cephadm 2023-07-10T17:00:01.291898+0000 mgr.y (mgr.14144) 187 : cephadm [INF] Saving service prometheus spec with placement smithi192=a;count:1 2023-07-10T17:00:02.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:02 smithi192 bash[23730]: audit 2023-07-10T17:00:01.299026+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:02.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:02 smithi192 bash[23730]: audit 2023-07-10T17:00:01.300651+0000 mon.a (mon.0) 468 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:00:02.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:02 smithi192 bash[23730]: audit 2023-07-10T17:00:01.309312+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:02.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:02 smithi192 bash[23730]: cephadm 2023-07-10T17:00:01.327470+0000 mgr.y (mgr.14144) 188 : cephadm [INF] Deploying daemon prometheus.a on smithi192 2023-07-10T17:00:02.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:02 smithi192 bash[23730]: cluster 2023-07-10T17:00:01.421544+0000 mgr.y (mgr.14144) 189 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:02.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:02 smithi116 bash[28351]: audit 2023-07-10T17:00:01.290405+0000 mgr.y (mgr.14144) 186 : audit [DBG] from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:00:02.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:02 smithi116 bash[28351]: cephadm 2023-07-10T17:00:01.291898+0000 mgr.y (mgr.14144) 187 : cephadm [INF] Saving service prometheus spec with placement smithi192=a;count:1 2023-07-10T17:00:02.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:02 smithi116 bash[28351]: audit 2023-07-10T17:00:01.299026+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:02.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:02 smithi116 bash[28351]: audit 2023-07-10T17:00:01.300651+0000 mon.a (mon.0) 468 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:00:02.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:02 smithi116 bash[28351]: audit 2023-07-10T17:00:01.309312+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:02.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:02 smithi116 bash[28351]: cephadm 2023-07-10T17:00:01.327470+0000 mgr.y (mgr.14144) 188 : cephadm [INF] Deploying daemon prometheus.a on smithi192 2023-07-10T17:00:02.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:02 smithi116 bash[28351]: cluster 2023-07-10T17:00:01.421544+0000 mgr.y (mgr.14144) 189 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:02.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:02 smithi116 bash[23740]: audit 2023-07-10T17:00:01.290405+0000 mgr.y (mgr.14144) 186 : audit [DBG] from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:00:02.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:02 smithi116 bash[23740]: cephadm 2023-07-10T17:00:01.291898+0000 mgr.y (mgr.14144) 187 : cephadm [INF] Saving service prometheus spec with placement smithi192=a;count:1 2023-07-10T17:00:02.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:02 smithi116 bash[23740]: audit 2023-07-10T17:00:01.299026+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:02.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:02 smithi116 bash[23740]: audit 2023-07-10T17:00:01.300651+0000 mon.a (mon.0) 468 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:00:02.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:02 smithi116 bash[23740]: audit 2023-07-10T17:00:01.309312+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:02.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:02 smithi116 bash[23740]: cephadm 2023-07-10T17:00:01.327470+0000 mgr.y (mgr.14144) 188 : cephadm [INF] Deploying daemon prometheus.a on smithi192 2023-07-10T17:00:02.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:02 smithi116 bash[23740]: cluster 2023-07-10T17:00:01.421544+0000 mgr.y (mgr.14144) 189 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:04 smithi192 bash[23730]: cluster 2023-07-10T17:00:03.422154+0000 mgr.y (mgr.14144) 190 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:04.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:04 smithi116 bash[23740]: cluster 2023-07-10T17:00:03.422154+0000 mgr.y (mgr.14144) 190 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:04.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:04 smithi116 bash[28351]: cluster 2023-07-10T17:00:03.422154+0000 mgr.y (mgr.14144) 190 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:06.698 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:06 smithi192 bash[23730]: cluster 2023-07-10T17:00:05.422858+0000 mgr.y (mgr.14144) 191 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:06.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:06 smithi116 bash[23740]: cluster 2023-07-10T17:00:05.422858+0000 mgr.y (mgr.14144) 191 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:06.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:06 smithi116 bash[28351]: cluster 2023-07-10T17:00:05.422858+0000 mgr.y (mgr.14144) 191 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:07.146 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled node-exporter update... 2023-07-10T17:00:08.399 DEBUG:teuthology.orchestra.run.smithi116:node-exporter.a> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@node-exporter.a.service 2023-07-10T17:00:08.403 DEBUG:teuthology.orchestra.run.smithi192:node-exporter.b> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@node-exporter.b.service 2023-07-10T17:00:08.405 INFO:tasks.cephadm:Adding alertmanager.a on smithi116 2023-07-10T17:00:08.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph orch apply alertmanager '1;smithi116=a' 2023-07-10T17:00:08.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:08 smithi192 bash[23730]: audit 2023-07-10T17:00:07.134349+0000 mgr.y (mgr.14144) 192 : audit [DBG] from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi116=a;smithi192=b", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:00:08.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:08 smithi192 bash[23730]: cephadm 2023-07-10T17:00:07.135938+0000 mgr.y (mgr.14144) 193 : cephadm [INF] Saving service node-exporter spec with placement smithi116=a;smithi192=b;count:2 2023-07-10T17:00:08.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:08 smithi192 bash[23730]: audit 2023-07-10T17:00:07.143056+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:08.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:08 smithi192 bash[23730]: cluster 2023-07-10T17:00:07.423495+0000 mgr.y (mgr.14144) 194 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:08.413 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:08 smithi116 bash[23740]: audit 2023-07-10T17:00:07.134349+0000 mgr.y (mgr.14144) 192 : audit [DBG] from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi116=a;smithi192=b", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:00:08.414 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:08 smithi116 bash[23740]: cephadm 2023-07-10T17:00:07.135938+0000 mgr.y (mgr.14144) 193 : cephadm [INF] Saving service node-exporter spec with placement smithi116=a;smithi192=b;count:2 2023-07-10T17:00:08.414 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:08 smithi116 bash[23740]: audit 2023-07-10T17:00:07.143056+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:08.414 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:08 smithi116 bash[23740]: cluster 2023-07-10T17:00:07.423495+0000 mgr.y (mgr.14144) 194 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:08.415 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:08 smithi116 bash[28351]: audit 2023-07-10T17:00:07.134349+0000 mgr.y (mgr.14144) 192 : audit [DBG] from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi116=a;smithi192=b", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:00:08.415 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:08 smithi116 bash[28351]: cephadm 2023-07-10T17:00:07.135938+0000 mgr.y (mgr.14144) 193 : cephadm [INF] Saving service node-exporter spec with placement smithi116=a;smithi192=b;count:2 2023-07-10T17:00:08.415 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:08 smithi116 bash[28351]: audit 2023-07-10T17:00:07.143056+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:08.415 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:08 smithi116 bash[28351]: cluster 2023-07-10T17:00:07.423495+0000 mgr.y (mgr.14144) 194 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:08.514 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T17:00:09.146 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T17:00:10.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:10 smithi192 bash[23730]: cluster 2023-07-10T17:00:09.424118+0000 mgr.y (mgr.14144) 195 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:10.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:10 smithi116 bash[23740]: cluster 2023-07-10T17:00:09.424118+0000 mgr.y (mgr.14144) 195 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:10.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:10 smithi116 bash[28351]: cluster 2023-07-10T17:00:09.424118+0000 mgr.y (mgr.14144) 195 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:12.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:12 smithi192 bash[23730]: cluster 2023-07-10T17:00:11.424745+0000 mgr.y (mgr.14144) 196 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:12.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:12 smithi116 bash[23740]: cluster 2023-07-10T17:00:11.424745+0000 mgr.y (mgr.14144) 196 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:12.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:12 smithi116 bash[28351]: cluster 2023-07-10T17:00:11.424745+0000 mgr.y (mgr.14144) 196 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:13.312 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled alertmanager update... 2023-07-10T17:00:14.294 DEBUG:teuthology.orchestra.run.smithi116:alertmanager.a> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@alertmanager.a.service 2023-07-10T17:00:14.297 INFO:tasks.cephadm:Adding grafana.a on smithi192 2023-07-10T17:00:14.297 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph orch apply grafana '1;smithi192=a' 2023-07-10T17:00:14.475 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T17:00:14.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:14 smithi192 bash[23730]: audit 2023-07-10T17:00:13.300218+0000 mgr.y (mgr.14144) 197 : audit [DBG] from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi116=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:00:14.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:14 smithi192 bash[23730]: cephadm 2023-07-10T17:00:13.301817+0000 mgr.y (mgr.14144) 198 : cephadm [INF] Saving service alertmanager spec with placement smithi116=a;count:1 2023-07-10T17:00:14.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:14 smithi192 bash[23730]: audit 2023-07-10T17:00:13.309288+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:14.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:14 smithi192 bash[23730]: cluster 2023-07-10T17:00:13.425371+0000 mgr.y (mgr.14144) 199 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:14.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:14 smithi116 bash[23740]: audit 2023-07-10T17:00:13.300218+0000 mgr.y (mgr.14144) 197 : audit [DBG] from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi116=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:00:14.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:14 smithi116 bash[23740]: cephadm 2023-07-10T17:00:13.301817+0000 mgr.y (mgr.14144) 198 : cephadm [INF] Saving service alertmanager spec with placement smithi116=a;count:1 2023-07-10T17:00:14.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:14 smithi116 bash[23740]: audit 2023-07-10T17:00:13.309288+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:14.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:14 smithi116 bash[23740]: cluster 2023-07-10T17:00:13.425371+0000 mgr.y (mgr.14144) 199 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:14.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:14 smithi116 bash[28351]: audit 2023-07-10T17:00:13.300218+0000 mgr.y (mgr.14144) 197 : audit [DBG] from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi116=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:00:14.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:14 smithi116 bash[28351]: cephadm 2023-07-10T17:00:13.301817+0000 mgr.y (mgr.14144) 198 : cephadm [INF] Saving service alertmanager spec with placement smithi116=a;count:1 2023-07-10T17:00:14.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:14 smithi116 bash[28351]: audit 2023-07-10T17:00:13.309288+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:14.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:14 smithi116 bash[28351]: cluster 2023-07-10T17:00:13.425371+0000 mgr.y (mgr.14144) 199 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:16.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:16 smithi192 bash[23730]: cluster 2023-07-10T17:00:15.426001+0000 mgr.y (mgr.14144) 200 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:16.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:16 smithi116 bash[23740]: cluster 2023-07-10T17:00:15.426001+0000 mgr.y (mgr.14144) 200 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:16.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:16 smithi116 bash[28351]: cluster 2023-07-10T17:00:15.426001+0000 mgr.y (mgr.14144) 200 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:18.785 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:18 smithi192 bash[23730]: cluster 2023-07-10T17:00:17.426656+0000 mgr.y (mgr.14144) 201 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:18.785 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:18 smithi192 bash[23730]: audit 2023-07-10T17:00:17.738053+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:18.786 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:18 smithi192 bash[23730]: audit 2023-07-10T17:00:17.739792+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:00:18.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:18 smithi116 bash[23740]: cluster 2023-07-10T17:00:17.426656+0000 mgr.y (mgr.14144) 201 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:18.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:18 smithi116 bash[23740]: audit 2023-07-10T17:00:17.738053+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:18.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:18 smithi116 bash[23740]: audit 2023-07-10T17:00:17.739792+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:00:18.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:18 smithi116 bash[28351]: cluster 2023-07-10T17:00:17.426656+0000 mgr.y (mgr.14144) 201 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:18.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:18 smithi116 bash[28351]: audit 2023-07-10T17:00:17.738053+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:18.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:18 smithi116 bash[28351]: audit 2023-07-10T17:00:17.739792+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:00:19.061 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:00:19 smithi192 bash[36820]: level=info ts=2023-07-10T17:00:19.057Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-10T17:00:19.061 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:00:19 smithi192 bash[36820]: level=info ts=2023-07-10T17:00:19.058Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-10T17:00:19.061 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:00:19 smithi192 bash[36820]: level=info ts=2023-07-10T17:00:19.058Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-10T17:00:19.217 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled grafana update... 2023-07-10T17:00:19.386 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:00:19 smithi192 bash[36820]: level=info ts=2023-07-10T17:00:19.062Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-10T17:00:19.386 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:00:19 smithi192 bash[36820]: level=info ts=2023-07-10T17:00:19.064Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2023-07-10T17:00:19.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:00:19 smithi192 bash[36820]: level=info ts=2023-07-10T17:00:19.064Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=1.901286ms 2023-07-10T17:00:19.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:00:19 smithi192 bash[36820]: level=info ts=2023-07-10T17:00:19.066Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-10T17:00:19.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:00:19 smithi192 bash[36820]: level=info ts=2023-07-10T17:00:19.066Z caller=main.go:695 msg="TSDB started" 2023-07-10T17:00:19.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:00:19 smithi192 bash[36820]: level=info ts=2023-07-10T17:00:19.066Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-10T17:00:19.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:00:19 smithi192 bash[36820]: level=info ts=2023-07-10T17:00:19.066Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-10T17:00:19.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:00:19 smithi192 bash[36820]: level=info ts=2023-07-10T17:00:19.067Z caller=main.go:646 msg="Server is ready to receive web requests." 2023-07-10T17:00:19.898 DEBUG:teuthology.orchestra.run.smithi192:grafana.a> sudo journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@grafana.a.service 2023-07-10T17:00:19.900 INFO:tasks.cephadm:Setting up client nodes... 2023-07-10T17:00:19.901 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-07-10T17:00:20.115 INFO:journalctl@ceph.grafana.a.smithi192.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-07-10T17:00:20.221 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:20 smithi116 bash[23740]: audit 2023-07-10T17:00:19.205202+0000 mgr.y (mgr.14144) 202 : audit [DBG] from='client.24380 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:00:20.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:20 smithi116 bash[23740]: cephadm 2023-07-10T17:00:19.206628+0000 mgr.y (mgr.14144) 203 : cephadm [INF] Saving service grafana spec with placement smithi192=a;count:1 2023-07-10T17:00:20.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:20 smithi116 bash[23740]: audit 2023-07-10T17:00:19.213985+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:20.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:20 smithi116 bash[23740]: cluster 2023-07-10T17:00:19.427267+0000 mgr.y (mgr.14144) 204 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:20.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:20 smithi116 bash[23740]: audit 2023-07-10T17:00:19.567270+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:00:20.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:20 smithi116 bash[23740]: audit 2023-07-10T17:00:19.570842+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:00:20.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:20 smithi116 bash[28351]: audit 2023-07-10T17:00:19.205202+0000 mgr.y (mgr.14144) 202 : audit [DBG] from='client.24380 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:00:20.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:20 smithi116 bash[28351]: cephadm 2023-07-10T17:00:19.206628+0000 mgr.y (mgr.14144) 203 : cephadm [INF] Saving service grafana spec with placement smithi192=a;count:1 2023-07-10T17:00:20.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:20 smithi116 bash[28351]: audit 2023-07-10T17:00:19.213985+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:20.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:20 smithi116 bash[28351]: cluster 2023-07-10T17:00:19.427267+0000 mgr.y (mgr.14144) 204 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:20.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:20 smithi116 bash[28351]: audit 2023-07-10T17:00:19.567270+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:00:20.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:20 smithi116 bash[28351]: audit 2023-07-10T17:00:19.570842+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:00:20.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:20 smithi192 bash[23730]: audit 2023-07-10T17:00:19.205202+0000 mgr.y (mgr.14144) 202 : audit [DBG] from='client.24380 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:00:20.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:20 smithi192 bash[23730]: cephadm 2023-07-10T17:00:19.206628+0000 mgr.y (mgr.14144) 203 : cephadm [INF] Saving service grafana spec with placement smithi192=a;count:1 2023-07-10T17:00:20.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:20 smithi192 bash[23730]: audit 2023-07-10T17:00:19.213985+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:20.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:20 smithi192 bash[23730]: cluster 2023-07-10T17:00:19.427267+0000 mgr.y (mgr.14144) 204 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:20.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:20 smithi192 bash[23730]: audit 2023-07-10T17:00:19.567270+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:00:20.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:20 smithi192 bash[23730]: audit 2023-07-10T17:00:19.570842+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:00:22.736 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:22 smithi116 bash[28351]: cluster 2023-07-10T17:00:21.427844+0000 mgr.y (mgr.14144) 205 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:22.737 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:22 smithi116 bash[28351]: audit 2023-07-10T17:00:22.212036+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:22.737 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:22 smithi116 bash[28351]: audit 2023-07-10T17:00:22.460198+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:22.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:22 smithi116 bash[23740]: cluster 2023-07-10T17:00:21.427844+0000 mgr.y (mgr.14144) 205 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:22.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:22 smithi116 bash[23740]: audit 2023-07-10T17:00:22.212036+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:22.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:22 smithi116 bash[23740]: audit 2023-07-10T17:00:22.460198+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:22.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:22 smithi192 bash[23730]: cluster 2023-07-10T17:00:21.427844+0000 mgr.y (mgr.14144) 205 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:22.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:22 smithi192 bash[23730]: audit 2023-07-10T17:00:22.212036+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:22.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:22 smithi192 bash[23730]: audit 2023-07-10T17:00:22.460198+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:23.182 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2023-07-10T17:00:23.183 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAnOaxkMTRZChAAi+QUbcBi3aLVDmk7Baxdmg== 2023-07-10T17:00:23.195 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:23 smithi192 systemd[1]: Started Ceph node-exporter.b for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:00:23.490 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:23 smithi192 bash[37296]: Unable to find image 'prom/node-exporter:v0.18.1' locally 2023-07-10T17:00:23.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:23 smithi116 bash[23740]: cephadm 2023-07-10T17:00:22.462946+0000 mgr.y (mgr.14144) 206 : cephadm [INF] Deploying daemon node-exporter.b on smithi192 2023-07-10T17:00:23.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:23 smithi116 bash[23740]: audit 2023-07-10T17:00:23.172532+0000 mon.c (mon.1) 17 : audit [INF] from='client.? 172.21.15.116:0/2718503987' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-10T17:00:23.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:23 smithi116 bash[23740]: audit 2023-07-10T17:00:23.173277+0000 mon.a (mon.0) 479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-10T17:00:23.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:23 smithi116 bash[23740]: audit 2023-07-10T17:00:23.179981+0000 mon.a (mon.0) 480 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-10T17:00:23.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:23 smithi116 bash[23740]: audit 2023-07-10T17:00:23.187993+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:23.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:23 smithi116 bash[28351]: cephadm 2023-07-10T17:00:22.462946+0000 mgr.y (mgr.14144) 206 : cephadm [INF] Deploying daemon node-exporter.b on smithi192 2023-07-10T17:00:23.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:23 smithi116 bash[28351]: audit 2023-07-10T17:00:23.172532+0000 mon.c (mon.1) 17 : audit [INF] from='client.? 172.21.15.116:0/2718503987' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-10T17:00:23.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:23 smithi116 bash[28351]: audit 2023-07-10T17:00:23.173277+0000 mon.a (mon.0) 479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-10T17:00:23.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:23 smithi116 bash[28351]: audit 2023-07-10T17:00:23.179981+0000 mon.a (mon.0) 480 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-10T17:00:23.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:23 smithi116 bash[28351]: audit 2023-07-10T17:00:23.187993+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:23.801 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:23 smithi192 bash[37296]: v0.18.1: Pulling from prom/node-exporter 2023-07-10T17:00:23.802 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:23 smithi192 bash[23730]: cephadm 2023-07-10T17:00:22.462946+0000 mgr.y (mgr.14144) 206 : cephadm [INF] Deploying daemon node-exporter.b on smithi192 2023-07-10T17:00:23.802 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:23 smithi192 bash[23730]: audit 2023-07-10T17:00:23.172532+0000 mon.c (mon.1) 17 : audit [INF] from='client.? 172.21.15.116:0/2718503987' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-10T17:00:23.802 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:23 smithi192 bash[23730]: audit 2023-07-10T17:00:23.173277+0000 mon.a (mon.0) 479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-10T17:00:23.802 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:23 smithi192 bash[23730]: audit 2023-07-10T17:00:23.179981+0000 mon.a (mon.0) 480 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-10T17:00:23.802 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:23 smithi192 bash[23730]: audit 2023-07-10T17:00:23.187993+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:23.837 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-07-10T17:00:23.837 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-07-10T17:00:23.838 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-07-10T17:00:23.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-07-10T17:00:24.008 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:23 smithi116 systemd[1]: Started Ceph node-exporter.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:00:24.136 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:23 smithi192 bash[37296]: 49a2d53aa1af: Pulling fs layer 2023-07-10T17:00:24.136 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:23 smithi192 bash[37296]: 3589a6efd9ce: Pulling fs layer 2023-07-10T17:00:24.136 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:23 smithi192 bash[37296]: 190160031744: Pulling fs layer 2023-07-10T17:00:24.136 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:23 smithi192 bash[37296]: 3589a6efd9ce: Verifying Checksum 2023-07-10T17:00:24.137 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:23 smithi192 bash[37296]: 3589a6efd9ce: Download complete 2023-07-10T17:00:24.137 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:23 smithi192 bash[37296]: 49a2d53aa1af: Verifying Checksum 2023-07-10T17:00:24.137 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:23 smithi192 bash[37296]: 49a2d53aa1af: Download complete 2023-07-10T17:00:24.137 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:23 smithi192 bash[37296]: 190160031744: Verifying Checksum 2023-07-10T17:00:24.137 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:23 smithi192 bash[37296]: 190160031744: Download complete 2023-07-10T17:00:24.387 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[41416]: Unable to find image 'prom/node-exporter:v0.18.1' locally 2023-07-10T17:00:24.494 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:24 smithi192 bash[37296]: 49a2d53aa1af: Pull complete 2023-07-10T17:00:24.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[23740]: cephadm 2023-07-10T17:00:23.189273+0000 mgr.y (mgr.14144) 207 : cephadm [INF] Deploying daemon node-exporter.a on smithi116 2023-07-10T17:00:24.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[23740]: cluster 2023-07-10T17:00:23.428399+0000 mgr.y (mgr.14144) 208 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:24.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[23740]: audit 2023-07-10T17:00:23.990853+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:24.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[28351]: cephadm 2023-07-10T17:00:23.189273+0000 mgr.y (mgr.14144) 207 : cephadm [INF] Deploying daemon node-exporter.a on smithi116 2023-07-10T17:00:24.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[28351]: cluster 2023-07-10T17:00:23.428399+0000 mgr.y (mgr.14144) 208 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:24.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[28351]: audit 2023-07-10T17:00:23.990853+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:24.644 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[41416]: v0.18.1: Pulling from prom/node-exporter 2023-07-10T17:00:24.644 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[41416]: 49a2d53aa1af: Pulling fs layer 2023-07-10T17:00:24.644 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[41416]: 3589a6efd9ce: Pulling fs layer 2023-07-10T17:00:24.644 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[41416]: 190160031744: Pulling fs layer 2023-07-10T17:00:24.644 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[41416]: 3589a6efd9ce: Download complete 2023-07-10T17:00:24.645 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[41416]: 49a2d53aa1af: Verifying Checksum 2023-07-10T17:00:24.645 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[41416]: 49a2d53aa1af: Download complete 2023-07-10T17:00:24.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:24 smithi192 bash[23730]: cephadm 2023-07-10T17:00:23.189273+0000 mgr.y (mgr.14144) 207 : cephadm [INF] Deploying daemon node-exporter.a on smithi116 2023-07-10T17:00:24.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:24 smithi192 bash[23730]: cluster 2023-07-10T17:00:23.428399+0000 mgr.y (mgr.14144) 208 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:24.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:24 smithi192 bash[23730]: audit 2023-07-10T17:00:23.990853+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:25.142 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[41416]: 190160031744: Verifying Checksum 2023-07-10T17:00:25.142 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[41416]: 190160031744: Download complete 2023-07-10T17:00:25.142 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:24 smithi116 bash[41416]: 49a2d53aa1af: Pull complete 2023-07-10T17:00:25.386 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:25 smithi192 bash[37296]: 3589a6efd9ce: Pull complete 2023-07-10T17:00:25.767 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:25 smithi192 bash[23730]: cephadm 2023-07-10T17:00:24.000584+0000 mgr.y (mgr.14144) 209 : cephadm [INF] Deploying daemon alertmanager.a on smithi116 2023-07-10T17:00:25.853 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:25 smithi116 bash[23740]: cephadm 2023-07-10T17:00:24.000584+0000 mgr.y (mgr.14144) 209 : cephadm [INF] Deploying daemon alertmanager.a on smithi116 2023-07-10T17:00:25.854 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:25 smithi116 bash[28351]: cephadm 2023-07-10T17:00:24.000584+0000 mgr.y (mgr.14144) 209 : cephadm [INF] Deploying daemon alertmanager.a on smithi116 2023-07-10T17:00:25.854 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:25 smithi116 bash[41416]: 3589a6efd9ce: Pull complete 2023-07-10T17:00:26.058 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:25 smithi192 bash[37296]: 190160031744: Pull complete 2023-07-10T17:00:26.059 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:26 smithi192 bash[37296]: Digest: sha256:a2f29256e53cc3e0b64d7a472512600b2e9410347d53cdc85b49f659c17e02ee 2023-07-10T17:00:26.142 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:25 smithi116 bash[41416]: 190160031744: Pull complete 2023-07-10T17:00:26.142 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:26 smithi116 bash[41416]: Digest: sha256:a2f29256e53cc3e0b64d7a472512600b2e9410347d53cdc85b49f659c17e02ee 2023-07-10T17:00:26.142 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:26 smithi116 bash[41416]: Status: Downloaded newer image for prom/node-exporter:v0.18.1 2023-07-10T17:00:26.386 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:26 smithi192 bash[37296]: Status: Downloaded newer image for prom/node-exporter:v0.18.1 2023-07-10T17:00:26.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:26 smithi192 bash[23730]: cluster 2023-07-10T17:00:25.429021+0000 mgr.y (mgr.14144) 210 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:26.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:26 smithi116 bash[23740]: cluster 2023-07-10T17:00:25.429021+0000 mgr.y (mgr.14144) 210 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:26.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:26 smithi116 bash[28351]: cluster 2023-07-10T17:00:25.429021+0000 mgr.y (mgr.14144) 210 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:28.386 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2023-07-10T17:00:28.387 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2023-07-10T17:00:28.387 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2023-07-10T17:00:28.387 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - arp" source="node_exporter.go:104" 2023-07-10T17:00:28.387 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - bcache" source="node_exporter.go:104" 2023-07-10T17:00:28.387 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - bonding" source="node_exporter.go:104" 2023-07-10T17:00:28.387 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2023-07-10T17:00:28.387 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - cpu" source="node_exporter.go:104" 2023-07-10T17:00:28.388 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2023-07-10T17:00:28.388 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2023-07-10T17:00:28.388 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - edac" source="node_exporter.go:104" 2023-07-10T17:00:28.388 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - entropy" source="node_exporter.go:104" 2023-07-10T17:00:28.388 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - filefd" source="node_exporter.go:104" 2023-07-10T17:00:28.388 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2023-07-10T17:00:28.388 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2023-07-10T17:00:28.389 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2023-07-10T17:00:28.389 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2023-07-10T17:00:28.389 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2023-07-10T17:00:28.389 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2023-07-10T17:00:28.389 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2023-07-10T17:00:28.389 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - netclass" source="node_exporter.go:104" 2023-07-10T17:00:28.390 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - netdev" source="node_exporter.go:104" 2023-07-10T17:00:28.390 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - netstat" source="node_exporter.go:104" 2023-07-10T17:00:28.390 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - nfs" source="node_exporter.go:104" 2023-07-10T17:00:28.390 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2023-07-10T17:00:28.391 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - pressure" source="node_exporter.go:104" 2023-07-10T17:00:28.391 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2023-07-10T17:00:28.391 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - stat" source="node_exporter.go:104" 2023-07-10T17:00:28.392 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - textfile" source="node_exporter.go:104" 2023-07-10T17:00:28.392 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - time" source="node_exporter.go:104" 2023-07-10T17:00:28.392 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - uname" source="node_exporter.go:104" 2023-07-10T17:00:28.392 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2023-07-10T17:00:28.393 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - xfs" source="node_exporter.go:104" 2023-07-10T17:00:28.393 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg=" - zfs" source="node_exporter.go:104" 2023-07-10T17:00:28.393 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[37296]: time="2023-07-10T17:00:28Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2023-07-10T17:00:28.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:28 smithi192 bash[23730]: cluster 2023-07-10T17:00:27.429653+0000 mgr.y (mgr.14144) 211 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:28.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[23740]: cluster 2023-07-10T17:00:27.429653+0000 mgr.y (mgr.14144) 211 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:28.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[28351]: cluster 2023-07-10T17:00:27.429653+0000 mgr.y (mgr.14144) 211 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:28.894 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2023-07-10T17:00:28.894 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2023-07-10T17:00:28.894 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2023-07-10T17:00:28.895 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - arp" source="node_exporter.go:104" 2023-07-10T17:00:28.895 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - bcache" source="node_exporter.go:104" 2023-07-10T17:00:28.895 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - bonding" source="node_exporter.go:104" 2023-07-10T17:00:28.895 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2023-07-10T17:00:28.895 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - cpu" source="node_exporter.go:104" 2023-07-10T17:00:28.895 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2023-07-10T17:00:28.895 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2023-07-10T17:00:28.896 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - edac" source="node_exporter.go:104" 2023-07-10T17:00:28.896 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - entropy" source="node_exporter.go:104" 2023-07-10T17:00:28.896 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - filefd" source="node_exporter.go:104" 2023-07-10T17:00:28.896 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2023-07-10T17:00:28.896 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2023-07-10T17:00:28.896 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2023-07-10T17:00:28.896 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2023-07-10T17:00:28.896 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2023-07-10T17:00:28.897 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2023-07-10T17:00:28.897 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2023-07-10T17:00:28.897 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - netclass" source="node_exporter.go:104" 2023-07-10T17:00:28.897 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - netdev" source="node_exporter.go:104" 2023-07-10T17:00:28.897 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - netstat" source="node_exporter.go:104" 2023-07-10T17:00:28.897 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - nfs" source="node_exporter.go:104" 2023-07-10T17:00:28.897 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2023-07-10T17:00:28.898 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - pressure" source="node_exporter.go:104" 2023-07-10T17:00:28.898 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2023-07-10T17:00:28.898 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - stat" source="node_exporter.go:104" 2023-07-10T17:00:28.898 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - textfile" source="node_exporter.go:104" 2023-07-10T17:00:28.898 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - time" source="node_exporter.go:104" 2023-07-10T17:00:28.898 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - uname" source="node_exporter.go:104" 2023-07-10T17:00:28.898 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2023-07-10T17:00:28.898 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - xfs" source="node_exporter.go:104" 2023-07-10T17:00:28.899 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg=" - zfs" source="node_exporter.go:104" 2023-07-10T17:00:28.899 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:00:28 smithi116 bash[41416]: time="2023-07-10T17:00:28Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2023-07-10T17:00:28.899 INFO:teuthology.orchestra.run.smithi192.stdout:[client.1] 2023-07-10T17:00:28.899 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAsOaxkvB+UNBAAm2hN2OCwIrdNW5sScVPvGA== 2023-07-10T17:00:29.641 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-07-10T17:00:29.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2023-07-10T17:00:29.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-07-10T17:00:29.666 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-07-10T17:00:29.667 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-07-10T17:00:29.668 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph mgr dump --format=json 2023-07-10T17:00:29.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:29 smithi192 bash[23730]: audit 2023-07-10T17:00:28.880972+0000 mon.b (mon.2) 21 : audit [INF] from='client.? 172.21.15.192:0/3201758879' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-10T17:00:29.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:29 smithi192 bash[23730]: audit 2023-07-10T17:00:28.881756+0000 mon.a (mon.0) 483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-10T17:00:29.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:29 smithi192 bash[23730]: audit 2023-07-10T17:00:28.888648+0000 mon.a (mon.0) 484 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-10T17:00:29.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:29 smithi116 bash[28351]: audit 2023-07-10T17:00:28.880972+0000 mon.b (mon.2) 21 : audit [INF] from='client.? 172.21.15.192:0/3201758879' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-10T17:00:29.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:29 smithi116 bash[28351]: audit 2023-07-10T17:00:28.881756+0000 mon.a (mon.0) 483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-10T17:00:29.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:29 smithi116 bash[28351]: audit 2023-07-10T17:00:28.888648+0000 mon.a (mon.0) 484 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-10T17:00:29.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:29 smithi116 bash[23740]: audit 2023-07-10T17:00:28.880972+0000 mon.b (mon.2) 21 : audit [INF] from='client.? 172.21.15.192:0/3201758879' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-10T17:00:29.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:29 smithi116 bash[23740]: audit 2023-07-10T17:00:28.881756+0000 mon.a (mon.0) 483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-10T17:00:29.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:29 smithi116 bash[23740]: audit 2023-07-10T17:00:28.888648+0000 mon.a (mon.0) 484 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-10T17:00:30.611 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:30 smithi116 bash[23740]: cluster 2023-07-10T17:00:29.430267+0000 mgr.y (mgr.14144) 212 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:30.612 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:30 smithi116 bash[28351]: cluster 2023-07-10T17:00:29.430267+0000 mgr.y (mgr.14144) 212 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:30.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:30 smithi192 bash[23730]: cluster 2023-07-10T17:00:29.430267+0000 mgr.y (mgr.14144) 212 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:00:32.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:32 smithi192 bash[23730]: cluster 2023-07-10T17:00:31.430914+0000 mgr.y (mgr.14144) 213 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:32.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:32 smithi116 bash[28351]: cluster 2023-07-10T17:00:31.430914+0000 mgr.y (mgr.14144) 213 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:32.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:32 smithi116 bash[23740]: cluster 2023-07-10T17:00:31.430914+0000 mgr.y (mgr.14144) 213 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:34.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:34 smithi192 bash[23730]: cluster 2023-07-10T17:00:33.431399+0000 mgr.y (mgr.14144) 214 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:34.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:34 smithi192 bash[23730]: audit 2023-07-10T17:00:34.044420+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:34.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:34 smithi192 bash[23730]: audit 2023-07-10T17:00:34.158823+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:34.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:34 smithi192 bash[23730]: audit 2023-07-10T17:00:34.168145+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:34.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:34 smithi192 bash[23730]: audit 2023-07-10T17:00:34.169102+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-10T17:00:34.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:34 smithi192 bash[23730]: audit 2023-07-10T17:00:34.176237+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:34.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:34 smithi116 bash[23740]: cluster 2023-07-10T17:00:33.431399+0000 mgr.y (mgr.14144) 214 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:34.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:34 smithi116 bash[23740]: audit 2023-07-10T17:00:34.044420+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:34.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:34 smithi116 bash[23740]: audit 2023-07-10T17:00:34.158823+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:34.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:34 smithi116 bash[23740]: audit 2023-07-10T17:00:34.168145+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:34.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:34 smithi116 bash[23740]: audit 2023-07-10T17:00:34.169102+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-10T17:00:34.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:34 smithi116 bash[23740]: audit 2023-07-10T17:00:34.176237+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:34.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:34 smithi116 bash[28351]: cluster 2023-07-10T17:00:33.431399+0000 mgr.y (mgr.14144) 214 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:34.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:34 smithi116 bash[28351]: audit 2023-07-10T17:00:34.044420+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:34.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:34 smithi116 bash[28351]: audit 2023-07-10T17:00:34.158823+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:34.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:34 smithi116 bash[28351]: audit 2023-07-10T17:00:34.168145+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:34.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:34 smithi116 bash[28351]: audit 2023-07-10T17:00:34.169102+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-10T17:00:34.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:34 smithi116 bash[28351]: audit 2023-07-10T17:00:34.176237+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:35.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:35 smithi192 bash[23730]: audit 2023-07-10T17:00:34.169809+0000 mgr.y (mgr.14144) 215 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-10T17:00:35.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:35 smithi192 bash[23730]: cephadm 2023-07-10T17:00:34.183501+0000 mgr.y (mgr.14144) 216 : cephadm [INF] Deploying daemon grafana.a on smithi192 2023-07-10T17:00:35.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:35 smithi116 bash[23740]: audit 2023-07-10T17:00:34.169809+0000 mgr.y (mgr.14144) 215 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-10T17:00:35.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:35 smithi116 bash[23740]: cephadm 2023-07-10T17:00:34.183501+0000 mgr.y (mgr.14144) 216 : cephadm [INF] Deploying daemon grafana.a on smithi192 2023-07-10T17:00:35.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:35 smithi116 bash[28351]: audit 2023-07-10T17:00:34.169809+0000 mgr.y (mgr.14144) 215 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-10T17:00:35.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:35 smithi116 bash[28351]: cephadm 2023-07-10T17:00:34.183501+0000 mgr.y (mgr.14144) 216 : cephadm [INF] Deploying daemon grafana.a on smithi192 2023-07-10T17:00:36.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:36 smithi116 bash[28351]: cluster 2023-07-10T17:00:35.432018+0000 mgr.y (mgr.14144) 217 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:36.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:36 smithi116 bash[23740]: cluster 2023-07-10T17:00:35.432018+0000 mgr.y (mgr.14144) 217 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:36.643 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:00:36 smithi116 bash[41991]: level=info ts=2023-07-10T17:00:36.509Z caller=main.go:497 msg=Listening address=:9093 2023-07-10T17:00:36.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:36 smithi192 bash[23730]: cluster 2023-07-10T17:00:35.432018+0000 mgr.y (mgr.14144) 217 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:37.090 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-07-10T17:00:37.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:37 smithi116 bash[23740]: audit 2023-07-10T17:00:37.079303+0000 mon.c (mon.1) 18 : audit [DBG] from='client.? 172.21.15.116:0/2799638310' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-10T17:00:37.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:37 smithi116 bash[28351]: audit 2023-07-10T17:00:37.079303+0000 mon.c (mon.1) 18 : audit [DBG] from='client.? 172.21.15.116:0/2799638310' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-10T17:00:37.809 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"active_gid":14144,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":670271520},{"type":"v1","addr":"172.21.15.116:6801","nonce":670271520}]},"active_addr":"172.21.15.116:6801/670271520","active_change":"2023-07-10T16:55:18.481603+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":24113,"name":"x","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://smithi116.front.sepia.ceph.com:8443/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":3314203798}]},{"addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":1335287284}]}]}} 2023-07-10T17:00:37.816 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-07-10T17:00:37.816 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-07-10T17:00:37.816 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph osd dump --format=json 2023-07-10T17:00:37.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:37 smithi192 bash[23730]: audit 2023-07-10T17:00:37.079303+0000 mon.c (mon.1) 18 : audit [DBG] from='client.? 172.21.15.116:0/2799638310' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-10T17:00:38.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:38 smithi192 bash[23730]: cluster 2023-07-10T17:00:37.432647+0000 mgr.y (mgr.14144) 218 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:38.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:38 smithi116 bash[23740]: cluster 2023-07-10T17:00:37.432647+0000 mgr.y (mgr.14144) 218 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:38.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:38 smithi116 bash[28351]: cluster 2023-07-10T17:00:37.432647+0000 mgr.y (mgr.14144) 218 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:38.892 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:00:38 smithi116 bash[41991]: level=info ts=2023-07-10T17:00:38.464Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000113062s 2023-07-10T17:00:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:00:40.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:40 smithi192 bash[23730]: cluster 2023-07-10T17:00:39.433296+0000 mgr.y (mgr.14144) 219 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:40.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:40 smithi116 bash[28351]: cluster 2023-07-10T17:00:39.433296+0000 mgr.y (mgr.14144) 219 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:40.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:40 smithi116 bash[23740]: cluster 2023-07-10T17:00:39.433296+0000 mgr.y (mgr.14144) 219 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:42.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:42 smithi192 bash[23730]: cluster 2023-07-10T17:00:41.433958+0000 mgr.y (mgr.14144) 220 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:42.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:42 smithi116 bash[23740]: cluster 2023-07-10T17:00:41.433958+0000 mgr.y (mgr.14144) 220 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:42.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:42 smithi116 bash[28351]: cluster 2023-07-10T17:00:41.433958+0000 mgr.y (mgr.14144) 220 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:43.491 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-07-10T17:00:43.492 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":51,"fsid":"3f83c2ec-1f42-11ee-9b31-001a4aab830c","created":"2023-07-10T16:54:21.709029+0000","modified":"2023-07-10T16:59:41.458155+0000","last_up_change":"2023-07-10T16:59:39.452978+0000","last_in_change":"2023-07-10T16:59:19.984246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-07-10T16:57:56.390654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"9234f7ee-8413-46af-98b7-6b957be6dd6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2297482263},{"type":"v1","addr":"172.21.15.116:6803","nonce":2297482263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2297482263},{"type":"v1","addr":"172.21.15.116:6805","nonce":2297482263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2297482263},{"type":"v1","addr":"172.21.15.116:6809","nonce":2297482263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2297482263},{"type":"v1","addr":"172.21.15.116:6807","nonce":2297482263}]},"public_addr":"172.21.15.116:6803/2297482263","cluster_addr":"172.21.15.116:6805/2297482263","heartbeat_back_addr":"172.21.15.116:6809/2297482263","heartbeat_front_addr":"172.21.15.116:6807/2297482263","state":["exists","up"]},{"osd":1,"uuid":"6d22b106-7d49-4c50-8c01-c018ae237563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3104401754},{"type":"v1","addr":"172.21.15.116:6811","nonce":3104401754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3104401754},{"type":"v1","addr":"172.21.15.116:6813","nonce":3104401754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":3104401754},{"type":"v1","addr":"172.21.15.116:6817","nonce":3104401754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3104401754},{"type":"v1","addr":"172.21.15.116:6815","nonce":3104401754}]},"public_addr":"172.21.15.116:6811/3104401754","cluster_addr":"172.21.15.116:6813/3104401754","heartbeat_back_addr":"172.21.15.116:6817/3104401754","heartbeat_front_addr":"172.21.15.116:6815/3104401754","state":["exists","up"]},{"osd":2,"uuid":"f1368d93-6cb3-4fe8-b945-974ee8982326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1907366872},{"type":"v1","addr":"172.21.15.116:6819","nonce":1907366872}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1907366872},{"type":"v1","addr":"172.21.15.116:6821","nonce":1907366872}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1907366872},{"type":"v1","addr":"172.21.15.116:6825","nonce":1907366872}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1907366872},{"type":"v1","addr":"172.21.15.116:6823","nonce":1907366872}]},"public_addr":"172.21.15.116:6819/1907366872","cluster_addr":"172.21.15.116:6821/1907366872","heartbeat_back_addr":"172.21.15.116:6825/1907366872","heartbeat_front_addr":"172.21.15.116:6823/1907366872","state":["exists","up"]},{"osd":3,"uuid":"d02ea7dd-deab-4a14-b7d2-56e45f1e51e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":278136235},{"type":"v1","addr":"172.21.15.116:6827","nonce":278136235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":278136235},{"type":"v1","addr":"172.21.15.116:6829","nonce":278136235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":278136235},{"type":"v1","addr":"172.21.15.116:6833","nonce":278136235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":278136235},{"type":"v1","addr":"172.21.15.116:6831","nonce":278136235}]},"public_addr":"172.21.15.116:6827/278136235","cluster_addr":"172.21.15.116:6829/278136235","heartbeat_back_addr":"172.21.15.116:6833/278136235","heartbeat_front_addr":"172.21.15.116:6831/278136235","state":["exists","up"]},{"osd":4,"uuid":"de3dac78-358c-4e5d-ae57-445a54a5c29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":1383435807},{"type":"v1","addr":"172.21.15.192:6801","nonce":1383435807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":1383435807},{"type":"v1","addr":"172.21.15.192:6803","nonce":1383435807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":1383435807},{"type":"v1","addr":"172.21.15.192:6807","nonce":1383435807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":1383435807},{"type":"v1","addr":"172.21.15.192:6805","nonce":1383435807}]},"public_addr":"172.21.15.192:6801/1383435807","cluster_addr":"172.21.15.192:6803/1383435807","heartbeat_back_addr":"172.21.15.192:6807/1383435807","heartbeat_front_addr":"172.21.15.192:6805/1383435807","state":["exists","up"]},{"osd":5,"uuid":"14218aa2-8ec8-407f-879b-b4fc762177fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2450645600},{"type":"v1","addr":"172.21.15.192:6809","nonce":2450645600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2450645600},{"type":"v1","addr":"172.21.15.192:6811","nonce":2450645600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2450645600},{"type":"v1","addr":"172.21.15.192:6815","nonce":2450645600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2450645600},{"type":"v1","addr":"172.21.15.192:6813","nonce":2450645600}]},"public_addr":"172.21.15.192:6809/2450645600","cluster_addr":"172.21.15.192:6811/2450645600","heartbeat_back_addr":"172.21.15.192:6815/2450645600","heartbeat_front_addr":"172.21.15.192:6813/2450645600","state":["exists","up"]},{"osd":6,"uuid":"b279a6e4-a920-46c2-b19b-59131f5c68f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":425451514},{"type":"v1","addr":"172.21.15.192:6817","nonce":425451514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":425451514},{"type":"v1","addr":"172.21.15.192:6819","nonce":425451514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":425451514},{"type":"v1","addr":"172.21.15.192:6823","nonce":425451514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":425451514},{"type":"v1","addr":"172.21.15.192:6821","nonce":425451514}]},"public_addr":"172.21.15.192:6817/425451514","cluster_addr":"172.21.15.192:6819/425451514","heartbeat_back_addr":"172.21.15.192:6823/425451514","heartbeat_front_addr":"172.21.15.192:6821/425451514","state":["exists","up"]},{"osd":7,"uuid":"e80e7f2c-5c06-41dd-9e63-5ce0d78b58ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2832282377},{"type":"v1","addr":"172.21.15.192:6825","nonce":2832282377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2832282377},{"type":"v1","addr":"172.21.15.192:6827","nonce":2832282377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2832282377},{"type":"v1","addr":"172.21.15.192:6831","nonce":2832282377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2832282377},{"type":"v1","addr":"172.21.15.192:6829","nonce":2832282377}]},"public_addr":"172.21.15.192:6825/2832282377","cluster_addr":"172.21.15.192:6827/2832282377","heartbeat_back_addr":"172.21.15.192:6831/2832282377","heartbeat_front_addr":"172.21.15.192:6829/2832282377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:56:45.784382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:57:11.613844+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:57:39.816175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:58:03.754473+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:58:25.910824+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:58:50.215523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:59:14.648618+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:59:37.987601+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/1249272063":"2023-07-11T16:55:18.481033+0000","172.21.15.116:0/3507856810":"2023-07-11T16:55:18.481033+0000","172.21.15.116:0/221490838":"2023-07-11T16:55:18.481033+0000","172.21.15.116:6801/1249272063":"2023-07-11T16:55:18.481033+0000","172.21.15.116:0/234488842":"2023-07-11T16:54:49.149280+0000","172.21.15.116:6800/48925241":"2023-07-11T16:54:49.149280+0000","172.21.15.116:6801/48925241":"2023-07-11T16:54:49.149280+0000","172.21.15.116:0/1382514600":"2023-07-11T16:54:49.149280+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-10T17:00:44.224 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-07-10T17:00:44.225 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph osd dump --format=json 2023-07-10T17:00:44.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:44 smithi116 bash[23740]: cluster 2023-07-10T17:00:43.434566+0000 mgr.y (mgr.14144) 221 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:44.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:44 smithi116 bash[23740]: audit 2023-07-10T17:00:43.487539+0000 mon.c (mon.1) 19 : audit [DBG] from='client.? 172.21.15.116:0/2406711438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-10T17:00:44.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:44 smithi116 bash[28351]: cluster 2023-07-10T17:00:43.434566+0000 mgr.y (mgr.14144) 221 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:44.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:44 smithi116 bash[28351]: audit 2023-07-10T17:00:43.487539+0000 mon.c (mon.1) 19 : audit [DBG] from='client.? 172.21.15.116:0/2406711438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-10T17:00:44.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:44 smithi192 bash[23730]: cluster 2023-07-10T17:00:43.434566+0000 mgr.y (mgr.14144) 221 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:44.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:44 smithi192 bash[23730]: audit 2023-07-10T17:00:43.487539+0000 mon.c (mon.1) 19 : audit [DBG] from='client.? 172.21.15.116:0/2406711438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-10T17:00:46.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:46 smithi192 bash[23730]: cluster 2023-07-10T17:00:45.435221+0000 mgr.y (mgr.14144) 222 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:46.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:46 smithi116 bash[28351]: cluster 2023-07-10T17:00:45.435221+0000 mgr.y (mgr.14144) 222 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:46.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:46 smithi116 bash[23740]: cluster 2023-07-10T17:00:45.435221+0000 mgr.y (mgr.14144) 222 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:46.893 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:00:46 smithi116 bash[41991]: level=info ts=2023-07-10T17:00:46.465Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.001020869s 2023-07-10T17:00:46.969 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:00:48.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:48 smithi192 bash[23730]: cluster 2023-07-10T17:00:47.435821+0000 mgr.y (mgr.14144) 223 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:48.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:48 smithi116 bash[28351]: cluster 2023-07-10T17:00:47.435821+0000 mgr.y (mgr.14144) 223 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:48.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:48 smithi116 bash[23740]: cluster 2023-07-10T17:00:47.435821+0000 mgr.y (mgr.14144) 223 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:50.021 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-07-10T17:00:50.022 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":51,"fsid":"3f83c2ec-1f42-11ee-9b31-001a4aab830c","created":"2023-07-10T16:54:21.709029+0000","modified":"2023-07-10T16:59:41.458155+0000","last_up_change":"2023-07-10T16:59:39.452978+0000","last_in_change":"2023-07-10T16:59:19.984246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-07-10T16:57:56.390654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"9234f7ee-8413-46af-98b7-6b957be6dd6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2297482263},{"type":"v1","addr":"172.21.15.116:6803","nonce":2297482263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2297482263},{"type":"v1","addr":"172.21.15.116:6805","nonce":2297482263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2297482263},{"type":"v1","addr":"172.21.15.116:6809","nonce":2297482263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2297482263},{"type":"v1","addr":"172.21.15.116:6807","nonce":2297482263}]},"public_addr":"172.21.15.116:6803/2297482263","cluster_addr":"172.21.15.116:6805/2297482263","heartbeat_back_addr":"172.21.15.116:6809/2297482263","heartbeat_front_addr":"172.21.15.116:6807/2297482263","state":["exists","up"]},{"osd":1,"uuid":"6d22b106-7d49-4c50-8c01-c018ae237563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3104401754},{"type":"v1","addr":"172.21.15.116:6811","nonce":3104401754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3104401754},{"type":"v1","addr":"172.21.15.116:6813","nonce":3104401754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":3104401754},{"type":"v1","addr":"172.21.15.116:6817","nonce":3104401754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3104401754},{"type":"v1","addr":"172.21.15.116:6815","nonce":3104401754}]},"public_addr":"172.21.15.116:6811/3104401754","cluster_addr":"172.21.15.116:6813/3104401754","heartbeat_back_addr":"172.21.15.116:6817/3104401754","heartbeat_front_addr":"172.21.15.116:6815/3104401754","state":["exists","up"]},{"osd":2,"uuid":"f1368d93-6cb3-4fe8-b945-974ee8982326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1907366872},{"type":"v1","addr":"172.21.15.116:6819","nonce":1907366872}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1907366872},{"type":"v1","addr":"172.21.15.116:6821","nonce":1907366872}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1907366872},{"type":"v1","addr":"172.21.15.116:6825","nonce":1907366872}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1907366872},{"type":"v1","addr":"172.21.15.116:6823","nonce":1907366872}]},"public_addr":"172.21.15.116:6819/1907366872","cluster_addr":"172.21.15.116:6821/1907366872","heartbeat_back_addr":"172.21.15.116:6825/1907366872","heartbeat_front_addr":"172.21.15.116:6823/1907366872","state":["exists","up"]},{"osd":3,"uuid":"d02ea7dd-deab-4a14-b7d2-56e45f1e51e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":278136235},{"type":"v1","addr":"172.21.15.116:6827","nonce":278136235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":278136235},{"type":"v1","addr":"172.21.15.116:6829","nonce":278136235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":278136235},{"type":"v1","addr":"172.21.15.116:6833","nonce":278136235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":278136235},{"type":"v1","addr":"172.21.15.116:6831","nonce":278136235}]},"public_addr":"172.21.15.116:6827/278136235","cluster_addr":"172.21.15.116:6829/278136235","heartbeat_back_addr":"172.21.15.116:6833/278136235","heartbeat_front_addr":"172.21.15.116:6831/278136235","state":["exists","up"]},{"osd":4,"uuid":"de3dac78-358c-4e5d-ae57-445a54a5c29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":1383435807},{"type":"v1","addr":"172.21.15.192:6801","nonce":1383435807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":1383435807},{"type":"v1","addr":"172.21.15.192:6803","nonce":1383435807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":1383435807},{"type":"v1","addr":"172.21.15.192:6807","nonce":1383435807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":1383435807},{"type":"v1","addr":"172.21.15.192:6805","nonce":1383435807}]},"public_addr":"172.21.15.192:6801/1383435807","cluster_addr":"172.21.15.192:6803/1383435807","heartbeat_back_addr":"172.21.15.192:6807/1383435807","heartbeat_front_addr":"172.21.15.192:6805/1383435807","state":["exists","up"]},{"osd":5,"uuid":"14218aa2-8ec8-407f-879b-b4fc762177fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2450645600},{"type":"v1","addr":"172.21.15.192:6809","nonce":2450645600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2450645600},{"type":"v1","addr":"172.21.15.192:6811","nonce":2450645600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2450645600},{"type":"v1","addr":"172.21.15.192:6815","nonce":2450645600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2450645600},{"type":"v1","addr":"172.21.15.192:6813","nonce":2450645600}]},"public_addr":"172.21.15.192:6809/2450645600","cluster_addr":"172.21.15.192:6811/2450645600","heartbeat_back_addr":"172.21.15.192:6815/2450645600","heartbeat_front_addr":"172.21.15.192:6813/2450645600","state":["exists","up"]},{"osd":6,"uuid":"b279a6e4-a920-46c2-b19b-59131f5c68f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":425451514},{"type":"v1","addr":"172.21.15.192:6817","nonce":425451514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":425451514},{"type":"v1","addr":"172.21.15.192:6819","nonce":425451514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":425451514},{"type":"v1","addr":"172.21.15.192:6823","nonce":425451514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":425451514},{"type":"v1","addr":"172.21.15.192:6821","nonce":425451514}]},"public_addr":"172.21.15.192:6817/425451514","cluster_addr":"172.21.15.192:6819/425451514","heartbeat_back_addr":"172.21.15.192:6823/425451514","heartbeat_front_addr":"172.21.15.192:6821/425451514","state":["exists","up"]},{"osd":7,"uuid":"e80e7f2c-5c06-41dd-9e63-5ce0d78b58ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2832282377},{"type":"v1","addr":"172.21.15.192:6825","nonce":2832282377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2832282377},{"type":"v1","addr":"172.21.15.192:6827","nonce":2832282377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2832282377},{"type":"v1","addr":"172.21.15.192:6831","nonce":2832282377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2832282377},{"type":"v1","addr":"172.21.15.192:6829","nonce":2832282377}]},"public_addr":"172.21.15.192:6825/2832282377","cluster_addr":"172.21.15.192:6827/2832282377","heartbeat_back_addr":"172.21.15.192:6831/2832282377","heartbeat_front_addr":"172.21.15.192:6829/2832282377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:56:45.784382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:57:11.613844+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:57:39.816175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:58:03.754473+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:58:25.910824+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:58:50.215523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:59:14.648618+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2023-07-10T16:59:37.987601+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/1249272063":"2023-07-11T16:55:18.481033+0000","172.21.15.116:0/3507856810":"2023-07-11T16:55:18.481033+0000","172.21.15.116:0/221490838":"2023-07-11T16:55:18.481033+0000","172.21.15.116:6801/1249272063":"2023-07-11T16:55:18.481033+0000","172.21.15.116:0/234488842":"2023-07-11T16:54:49.149280+0000","172.21.15.116:6800/48925241":"2023-07-11T16:54:49.149280+0000","172.21.15.116:6801/48925241":"2023-07-11T16:54:49.149280+0000","172.21.15.116:0/1382514600":"2023-07-11T16:54:49.149280+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-10T17:00:50.340 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:50 smithi192 systemd[1]: Started Ceph grafana.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:00:50.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:50 smithi192 bash[23730]: cluster 2023-07-10T17:00:49.436487+0000 mgr.y (mgr.14144) 224 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:50.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:50 smithi192 bash[23730]: audit 2023-07-10T17:00:50.017933+0000 mon.a (mon.0) 490 : audit [DBG] from='client.? 172.21.15.116:0/659856833' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-10T17:00:50.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:50 smithi192 bash[23730]: audit 2023-07-10T17:00:50.375579+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:50.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:50 smithi192 bash[23730]: audit 2023-07-10T17:00:50.377549+0000 mon.a (mon.0) 492 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:00:50.717 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:50 smithi116 bash[23740]: cluster 2023-07-10T17:00:49.436487+0000 mgr.y (mgr.14144) 224 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:50.717 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:50 smithi116 bash[23740]: audit 2023-07-10T17:00:50.017933+0000 mon.a (mon.0) 490 : audit [DBG] from='client.? 172.21.15.116:0/659856833' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-10T17:00:50.717 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:50 smithi116 bash[23740]: audit 2023-07-10T17:00:50.375579+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:50.717 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:50 smithi116 bash[23740]: audit 2023-07-10T17:00:50.377549+0000 mon.a (mon.0) 492 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:00:50.718 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:50 smithi116 bash[28351]: cluster 2023-07-10T17:00:49.436487+0000 mgr.y (mgr.14144) 224 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:50.718 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:50 smithi116 bash[28351]: audit 2023-07-10T17:00:50.017933+0000 mon.a (mon.0) 490 : audit [DBG] from='client.? 172.21.15.116:0/659856833' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-10T17:00:50.718 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:50 smithi116 bash[28351]: audit 2023-07-10T17:00:50.375579+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:50.718 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:50 smithi116 bash[28351]: audit 2023-07-10T17:00:50.377549+0000 mon.a (mon.0) 492 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:00:50.719 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2023-07-10T17:00:50.720 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2023-07-10T17:00:50.720 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2023-07-10T17:00:50.721 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2023-07-10T17:00:50.721 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2023-07-10T17:00:50.722 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2023-07-10T17:00:50.722 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2023-07-10T17:00:50.723 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2023-07-10T17:00:51.607 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Starting Grafana" logger=server version=6.7.4 commit=8e44bbc5f5 branch=HEAD compiled=2020-05-26T17:35:38+0000 2023-07-10T17:00:51.608 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2023-07-10T17:00:51.608 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2023-07-10T17:00:51.608 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2023-07-10T17:00:51.608 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2023-07-10T17:00:51.608 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2023-07-10T17:00:51.608 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2023-07-10T17:00:51.608 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2023-07-10T17:00:51.609 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2023-07-10T17:00:51.609 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2023-07-10T17:00:51.609 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2023-07-10T17:00:51.609 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2023-07-10T17:00:51.609 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="App mode production" logger=settings 2023-07-10T17:00:51.610 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Initializing SqlStore" logger=server 2023-07-10T17:00:51.610 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2023-07-10T17:00:51.610 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Starting DB migration" logger=migrator 2023-07-10T17:00:51.610 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2023-07-10T17:00:51.611 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2023-07-10T17:00:51.611 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2023-07-10T17:00:51.884 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2023-07-10T17:00:51.885 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2023-07-10T17:00:51.885 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2023-07-10T17:00:51.885 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2023-07-10T17:00:51.885 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2023-07-10T17:00:52.136 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:51 smithi192 bash[37887]: t=2023-07-10T17:00:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2023-07-10T17:00:52.136 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[37887]: t=2023-07-10T17:00:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2023-07-10T17:00:52.136 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[37887]: t=2023-07-10T17:00:52+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2023-07-10T17:00:52.430 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[37887]: t=2023-07-10T17:00:52+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2023-07-10T17:00:52.430 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[37887]: t=2023-07-10T17:00:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2023-07-10T17:00:52.430 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[37887]: t=2023-07-10T17:00:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2023-07-10T17:00:52.430 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[37887]: t=2023-07-10T17:00:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2023-07-10T17:00:52.431 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[37887]: t=2023-07-10T17:00:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2023-07-10T17:00:52.431 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[37887]: t=2023-07-10T17:00:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2023-07-10T17:00:52.690 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[23730]: cluster 2023-07-10T17:00:51.437118+0000 mgr.y (mgr.14144) 225 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:52.690 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[37887]: t=2023-07-10T17:00:52+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2023-07-10T17:00:52.690 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[37887]: t=2023-07-10T17:00:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2023-07-10T17:00:52.691 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[37887]: t=2023-07-10T17:00:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2023-07-10T17:00:52.691 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[37887]: t=2023-07-10T17:00:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2023-07-10T17:00:52.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:52 smithi116 bash[23740]: cluster 2023-07-10T17:00:51.437118+0000 mgr.y (mgr.14144) 225 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:52.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:52 smithi116 bash[28351]: cluster 2023-07-10T17:00:51.437118+0000 mgr.y (mgr.14144) 225 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:52.967 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[37887]: t=2023-07-10T17:00:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2023-07-10T17:00:52.967 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[37887]: t=2023-07-10T17:00:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2023-07-10T17:00:52.967 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[37887]: t=2023-07-10T17:00:52+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2023-07-10T17:00:53.142 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:00:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:00:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:00:53.144 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:00:53.144 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:00:53.144 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:00:53.147 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:00:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:00:53.303 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:52 smithi192 bash[37887]: t=2023-07-10T17:00:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2023-07-10T17:00:53.303 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:53 smithi192 bash[37887]: t=2023-07-10T17:00:53+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2023-07-10T17:00:53.303 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:53 smithi192 bash[37887]: t=2023-07-10T17:00:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2023-07-10T17:00:53.303 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:53 smithi192 bash[37887]: t=2023-07-10T17:00:53+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2023-07-10T17:00:53.613 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:53 smithi192 bash[37887]: t=2023-07-10T17:00:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2023-07-10T17:00:53.613 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:53 smithi192 bash[37887]: t=2023-07-10T17:00:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2023-07-10T17:00:53.614 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:53 smithi192 bash[37887]: t=2023-07-10T17:00:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2023-07-10T17:00:53.614 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:53 smithi192 bash[37887]: t=2023-07-10T17:00:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2023-07-10T17:00:53.614 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:53 smithi192 bash[37887]: t=2023-07-10T17:00:53+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2023-07-10T17:00:53.865 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:53 smithi192 bash[37887]: t=2023-07-10T17:00:53+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2023-07-10T17:00:53.865 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:53 smithi192 bash[37887]: t=2023-07-10T17:00:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2023-07-10T17:00:53.865 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:53 smithi192 bash[37887]: t=2023-07-10T17:00:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2023-07-10T17:00:54.125 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:53 smithi192 bash[37887]: t=2023-07-10T17:00:53+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2023-07-10T17:00:54.125 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:53 smithi192 bash[37887]: t=2023-07-10T17:00:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2023-07-10T17:00:54.126 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:54 smithi192 bash[37887]: t=2023-07-10T17:00:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2023-07-10T17:00:54.126 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:54 smithi192 bash[37887]: t=2023-07-10T17:00:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2023-07-10T17:00:54.386 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:54 smithi192 bash[37887]: t=2023-07-10T17:00:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2023-07-10T17:00:54.386 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:54 smithi192 bash[37887]: t=2023-07-10T17:00:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2023-07-10T17:00:54.386 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:54 smithi192 bash[37887]: t=2023-07-10T17:00:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2023-07-10T17:00:54.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:54 smithi192 bash[23730]: cluster 2023-07-10T17:00:53.437662+0000 mgr.y (mgr.14144) 226 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:54 smithi192 bash[23730]: audit 2023-07-10T17:00:54.263399+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:54.738 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:54 smithi192 bash[37887]: t=2023-07-10T17:00:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2023-07-10T17:00:54.738 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:54 smithi192 bash[37887]: t=2023-07-10T17:00:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2023-07-10T17:00:54.738 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:54 smithi192 bash[37887]: t=2023-07-10T17:00:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2023-07-10T17:00:54.738 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:54 smithi192 bash[37887]: t=2023-07-10T17:00:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2023-07-10T17:00:54.739 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:54 smithi192 bash[37887]: t=2023-07-10T17:00:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2023-07-10T17:00:54.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:54 smithi116 bash[28351]: cluster 2023-07-10T17:00:53.437662+0000 mgr.y (mgr.14144) 226 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:54.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:54 smithi116 bash[28351]: audit 2023-07-10T17:00:54.263399+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:54.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:54 smithi116 bash[23740]: cluster 2023-07-10T17:00:53.437662+0000 mgr.y (mgr.14144) 226 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:54.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:54 smithi116 bash[23740]: audit 2023-07-10T17:00:54.263399+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:55.014 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:54 smithi192 bash[37887]: t=2023-07-10T17:00:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2023-07-10T17:00:55.014 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:54 smithi192 bash[37887]: t=2023-07-10T17:00:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2023-07-10T17:00:55.014 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[37887]: t=2023-07-10T17:00:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2023-07-10T17:00:55.332 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[37887]: t=2023-07-10T17:00:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2023-07-10T17:00:55.333 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[37887]: t=2023-07-10T17:00:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2023-07-10T17:00:55.333 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[37887]: t=2023-07-10T17:00:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2023-07-10T17:00:55.333 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[37887]: t=2023-07-10T17:00:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2023-07-10T17:00:55.334 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[37887]: t=2023-07-10T17:00:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2023-07-10T17:00:55.601 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[37887]: t=2023-07-10T17:00:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2023-07-10T17:00:55.601 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[37887]: t=2023-07-10T17:00:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2023-07-10T17:00:55.601 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[37887]: t=2023-07-10T17:00:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2023-07-10T17:00:55.601 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[37887]: t=2023-07-10T17:00:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2023-07-10T17:00:55.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[23730]: audit 2023-07-10T17:00:54.819302+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:55.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[23730]: audit 2023-07-10T17:00:54.829013+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:55.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[23730]: cephadm 2023-07-10T17:00:54.832636+0000 mgr.y (mgr.14144) 227 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-07-10T17:00:55.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[23730]: cephadm 2023-07-10T17:00:54.836846+0000 mgr.y (mgr.14144) 228 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi116 2023-07-10T17:00:55.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[23730]: cluster 2023-07-10T17:00:55.438227+0000 mgr.y (mgr.14144) 229 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:55.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[37887]: t=2023-07-10T17:00:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2023-07-10T17:00:55.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[37887]: t=2023-07-10T17:00:55+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2023-07-10T17:00:55.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[37887]: t=2023-07-10T17:00:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2023-07-10T17:00:55.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[37887]: t=2023-07-10T17:00:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2023-07-10T17:00:55.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:55 smithi116 bash[28351]: audit 2023-07-10T17:00:54.819302+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:55.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:55 smithi116 bash[28351]: audit 2023-07-10T17:00:54.829013+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:55.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:55 smithi116 bash[28351]: cephadm 2023-07-10T17:00:54.832636+0000 mgr.y (mgr.14144) 227 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-07-10T17:00:55.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:55 smithi116 bash[28351]: cephadm 2023-07-10T17:00:54.836846+0000 mgr.y (mgr.14144) 228 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi116 2023-07-10T17:00:55.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:55 smithi116 bash[28351]: cluster 2023-07-10T17:00:55.438227+0000 mgr.y (mgr.14144) 229 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:55.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:55 smithi116 bash[23740]: audit 2023-07-10T17:00:54.819302+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:55.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:55 smithi116 bash[23740]: audit 2023-07-10T17:00:54.829013+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:00:55.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:55 smithi116 bash[23740]: cephadm 2023-07-10T17:00:54.832636+0000 mgr.y (mgr.14144) 227 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-07-10T17:00:55.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:55 smithi116 bash[23740]: cephadm 2023-07-10T17:00:54.836846+0000 mgr.y (mgr.14144) 228 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi116 2023-07-10T17:00:55.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:55 smithi116 bash[23740]: cluster 2023-07-10T17:00:55.438227+0000 mgr.y (mgr.14144) 229 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:56.196 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[37887]: t=2023-07-10T17:00:55+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2023-07-10T17:00:56.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:55 smithi192 bash[37887]: t=2023-07-10T17:00:55+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2023-07-10T17:00:56.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:56 smithi192 bash[37887]: t=2023-07-10T17:00:56+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2023-07-10T17:00:56.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:56 smithi192 bash[37887]: t=2023-07-10T17:00:56+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2023-07-10T17:00:56.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:56 smithi192 bash[37887]: t=2023-07-10T17:00:56+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2023-07-10T17:00:56.507 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:56 smithi192 bash[37887]: t=2023-07-10T17:00:56+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2023-07-10T17:00:56.507 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:56 smithi192 bash[37887]: t=2023-07-10T17:00:56+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2023-07-10T17:00:56.507 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:56 smithi192 bash[37887]: t=2023-07-10T17:00:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2023-07-10T17:00:56.507 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:56 smithi192 bash[37887]: t=2023-07-10T17:00:56+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2023-07-10T17:00:56.758 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:56 smithi192 bash[37887]: t=2023-07-10T17:00:56+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2023-07-10T17:00:56.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:56 smithi192 bash[37887]: t=2023-07-10T17:00:56+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2023-07-10T17:00:56.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:56 smithi192 bash[37887]: t=2023-07-10T17:00:56+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2023-07-10T17:00:56.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:56 smithi192 bash[37887]: t=2023-07-10T17:00:56+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2023-07-10T17:00:57.026 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:56 smithi192 bash[37887]: t=2023-07-10T17:00:56+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2023-07-10T17:00:57.027 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:56 smithi192 bash[37887]: t=2023-07-10T17:00:56+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2023-07-10T17:00:57.330 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:57 smithi192 bash[37887]: t=2023-07-10T17:00:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2023-07-10T17:00:57.331 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:57 smithi192 bash[37887]: t=2023-07-10T17:00:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2023-07-10T17:00:57.331 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:57 smithi192 bash[37887]: t=2023-07-10T17:00:57+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2023-07-10T17:00:57.331 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:57 smithi192 bash[37887]: t=2023-07-10T17:00:57+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2023-07-10T17:00:57.331 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:57 smithi192 bash[37887]: t=2023-07-10T17:00:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2023-07-10T17:00:57.621 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:57 smithi192 bash[37887]: t=2023-07-10T17:00:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2023-07-10T17:00:57.621 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:57 smithi192 bash[37887]: t=2023-07-10T17:00:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2023-07-10T17:00:57.622 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:57 smithi192 bash[37887]: t=2023-07-10T17:00:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2023-07-10T17:00:57.886 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:57 smithi192 bash[37887]: t=2023-07-10T17:00:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2023-07-10T17:00:57.886 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:57 smithi192 bash[37887]: t=2023-07-10T17:00:57+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2023-07-10T17:00:57.886 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:57 smithi192 bash[37887]: t=2023-07-10T17:00:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2023-07-10T17:00:57.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:57 smithi192 bash[37887]: t=2023-07-10T17:00:57+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2023-07-10T17:00:57.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:57 smithi192 bash[37887]: t=2023-07-10T17:00:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2023-07-10T17:00:58.183 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:57 smithi192 bash[37887]: t=2023-07-10T17:00:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2023-07-10T17:00:58.183 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:57 smithi192 bash[37887]: t=2023-07-10T17:00:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2023-07-10T17:00:58.184 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:58 smithi192 bash[37887]: t=2023-07-10T17:00:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2023-07-10T17:00:58.184 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:58 smithi192 bash[37887]: t=2023-07-10T17:00:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2023-07-10T17:00:58.460 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:58 smithi192 bash[37887]: t=2023-07-10T17:00:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2023-07-10T17:00:58.460 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:58 smithi192 bash[37887]: t=2023-07-10T17:00:58+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2023-07-10T17:00:58.460 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:58 smithi192 bash[37887]: t=2023-07-10T17:00:58+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2023-07-10T17:00:58.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:00:58 smithi116 bash[28351]: cluster 2023-07-10T17:00:57.438807+0000 mgr.y (mgr.14144) 230 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:58.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:00:58 smithi116 bash[23740]: cluster 2023-07-10T17:00:57.438807+0000 mgr.y (mgr.14144) 230 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:58.711 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:00:58 smithi192 bash[23730]: cluster 2023-07-10T17:00:57.438807+0000 mgr.y (mgr.14144) 230 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:00:58.712 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:58 smithi192 bash[37887]: t=2023-07-10T17:00:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2023-07-10T17:00:58.712 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:58 smithi192 bash[37887]: t=2023-07-10T17:00:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2023-07-10T17:00:58.712 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:58 smithi192 bash[37887]: t=2023-07-10T17:00:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2023-07-10T17:00:58.713 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:58 smithi192 bash[37887]: t=2023-07-10T17:00:58+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2023-07-10T17:00:58.713 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:58 smithi192 bash[37887]: t=2023-07-10T17:00:58+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2023-07-10T17:00:59.030 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:58 smithi192 bash[37887]: t=2023-07-10T17:00:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2023-07-10T17:00:59.031 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:58 smithi192 bash[37887]: t=2023-07-10T17:00:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2023-07-10T17:00:59.031 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:58 smithi192 bash[37887]: t=2023-07-10T17:00:58+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2023-07-10T17:00:59.031 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:59 smithi192 bash[37887]: t=2023-07-10T17:00:59+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2023-07-10T17:00:59.306 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:59 smithi192 bash[37887]: t=2023-07-10T17:00:59+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2023-07-10T17:00:59.307 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:59 smithi192 bash[37887]: t=2023-07-10T17:00:59+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2023-07-10T17:00:59.307 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:59 smithi192 bash[37887]: t=2023-07-10T17:00:59+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2023-07-10T17:00:59.575 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:59 smithi192 bash[37887]: t=2023-07-10T17:00:59+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2023-07-10T17:00:59.575 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:59 smithi192 bash[37887]: t=2023-07-10T17:00:59+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2023-07-10T17:00:59.575 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:59 smithi192 bash[37887]: t=2023-07-10T17:00:59+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2023-07-10T17:00:59.575 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:59 smithi192 bash[37887]: t=2023-07-10T17:00:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2023-07-10T17:00:59.576 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:59 smithi192 bash[37887]: t=2023-07-10T17:00:59+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2023-07-10T17:00:59.876 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:59 smithi192 bash[37887]: t=2023-07-10T17:00:59+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2023-07-10T17:00:59.877 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:59 smithi192 bash[37887]: t=2023-07-10T17:00:59+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2023-07-10T17:00:59.877 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:59 smithi192 bash[37887]: t=2023-07-10T17:00:59+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2023-07-10T17:01:00.135 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:59 smithi192 bash[37887]: t=2023-07-10T17:00:59+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2023-07-10T17:01:00.136 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:00:59 smithi192 bash[37887]: t=2023-07-10T17:00:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2023-07-10T17:01:00.136 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:00 smithi192 bash[37887]: t=2023-07-10T17:01:00+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2023-07-10T17:01:00.136 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:00 smithi192 bash[37887]: t=2023-07-10T17:01:00+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2023-07-10T17:01:00.397 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:00 smithi192 bash[37887]: t=2023-07-10T17:01:00+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2023-07-10T17:01:00.397 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:00 smithi192 bash[37887]: t=2023-07-10T17:01:00+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2023-07-10T17:01:00.397 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:00 smithi192 bash[37887]: t=2023-07-10T17:01:00+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2023-07-10T17:01:00.397 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:00 smithi192 bash[37887]: t=2023-07-10T17:01:00+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2023-07-10T17:01:00.397 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:00 smithi192 bash[37887]: t=2023-07-10T17:01:00+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2023-07-10T17:01:00.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:00 smithi116 bash[23740]: cluster 2023-07-10T17:00:59.439593+0000 mgr.y (mgr.14144) 231 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:00.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:00 smithi116 bash[28351]: cluster 2023-07-10T17:00:59.439593+0000 mgr.y (mgr.14144) 231 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:00.748 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:00 smithi192 bash[23730]: cluster 2023-07-10T17:00:59.439593+0000 mgr.y (mgr.14144) 231 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:00.749 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:00 smithi192 bash[37887]: t=2023-07-10T17:01:00+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2023-07-10T17:01:00.749 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:00 smithi192 bash[37887]: t=2023-07-10T17:01:00+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2023-07-10T17:01:00.749 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:00 smithi192 bash[37887]: t=2023-07-10T17:01:00+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2023-07-10T17:01:00.749 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:00 smithi192 bash[37887]: t=2023-07-10T17:01:00+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2023-07-10T17:01:01.008 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:00 smithi192 bash[37887]: t=2023-07-10T17:01:00+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2023-07-10T17:01:01.009 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:00 smithi192 bash[37887]: t=2023-07-10T17:01:00+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2023-07-10T17:01:01.009 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:00 smithi192 bash[37887]: t=2023-07-10T17:01:00+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2023-07-10T17:01:01.009 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:01 smithi192 bash[37887]: t=2023-07-10T17:01:01+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2023-07-10T17:01:01.259 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:01 smithi192 bash[37887]: t=2023-07-10T17:01:01+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2023-07-10T17:01:01.260 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:01 smithi192 bash[37887]: t=2023-07-10T17:01:01+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2023-07-10T17:01:01.553 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:01 smithi192 bash[37887]: t=2023-07-10T17:01:01+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2023-07-10T17:01:01.554 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:01 smithi192 bash[37887]: t=2023-07-10T17:01:01+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2023-07-10T17:01:01.554 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:01 smithi192 bash[37887]: t=2023-07-10T17:01:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2023-07-10T17:01:01.554 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:01 smithi192 bash[37887]: t=2023-07-10T17:01:01+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2023-07-10T17:01:01.554 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:01 smithi192 bash[37887]: t=2023-07-10T17:01:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2023-07-10T17:01:01.838 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:01 smithi192 bash[37887]: t=2023-07-10T17:01:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2023-07-10T17:01:01.838 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:01 smithi192 bash[37887]: t=2023-07-10T17:01:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2023-07-10T17:01:01.839 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:01 smithi192 bash[37887]: t=2023-07-10T17:01:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2023-07-10T17:01:01.839 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:01 smithi192 bash[37887]: t=2023-07-10T17:01:01+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2023-07-10T17:01:02.135 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:01 smithi192 bash[37887]: t=2023-07-10T17:01:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2023-07-10T17:01:02.136 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:01 smithi192 bash[37887]: t=2023-07-10T17:01:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2023-07-10T17:01:02.136 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:02 smithi192 bash[37887]: t=2023-07-10T17:01:02+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2023-07-10T17:01:02.416 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:02 smithi192 bash[37887]: t=2023-07-10T17:01:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2023-07-10T17:01:02.417 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:02 smithi192 bash[37887]: t=2023-07-10T17:01:02+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2023-07-10T17:01:02.417 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:02 smithi192 bash[37887]: t=2023-07-10T17:01:02+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2023-07-10T17:01:02.417 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:02 smithi192 bash[37887]: t=2023-07-10T17:01:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2023-07-10T17:01:02.417 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:02 smithi192 bash[37887]: t=2023-07-10T17:01:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2023-07-10T17:01:02.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:02 smithi192 bash[23730]: cluster 2023-07-10T17:01:01.440318+0000 mgr.y (mgr.14144) 232 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:02.668 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:02 smithi192 bash[37887]: t=2023-07-10T17:01:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2023-07-10T17:01:02.668 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:02 smithi192 bash[37887]: t=2023-07-10T17:01:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2023-07-10T17:01:02.669 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:02 smithi192 bash[37887]: t=2023-07-10T17:01:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2023-07-10T17:01:02.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:02 smithi116 bash[23740]: cluster 2023-07-10T17:01:01.440318+0000 mgr.y (mgr.14144) 232 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:02.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:02 smithi116 bash[28351]: cluster 2023-07-10T17:01:01.440318+0000 mgr.y (mgr.14144) 232 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:02.962 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:02 smithi192 bash[37887]: t=2023-07-10T17:01:02+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2023-07-10T17:01:02.962 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:02 smithi192 bash[37887]: t=2023-07-10T17:01:02+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2023-07-10T17:01:02.962 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:02 smithi192 bash[37887]: t=2023-07-10T17:01:02+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2023-07-10T17:01:02.962 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:02 smithi192 bash[37887]: t=2023-07-10T17:01:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2023-07-10T17:01:02.962 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:02 smithi192 bash[37887]: t=2023-07-10T17:01:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2023-07-10T17:01:03.230 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:03 smithi192 bash[37887]: t=2023-07-10T17:01:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2023-07-10T17:01:03.230 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:03 smithi192 bash[37887]: t=2023-07-10T17:01:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2023-07-10T17:01:03.230 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:03 smithi192 bash[37887]: t=2023-07-10T17:01:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2023-07-10T17:01:03.498 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:03 smithi192 bash[37887]: t=2023-07-10T17:01:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2023-07-10T17:01:03.498 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:03 smithi192 bash[37887]: t=2023-07-10T17:01:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2023-07-10T17:01:03.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:03 smithi192 bash[37887]: t=2023-07-10T17:01:03+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2023-07-10T17:01:03.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:03 smithi192 bash[37887]: t=2023-07-10T17:01:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2023-07-10T17:01:03.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:03 smithi192 bash[37887]: t=2023-07-10T17:01:03+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2023-07-10T17:01:03.783 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:03 smithi192 bash[37887]: t=2023-07-10T17:01:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2023-07-10T17:01:03.783 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:03 smithi192 bash[37887]: t=2023-07-10T17:01:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2023-07-10T17:01:03.783 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:03 smithi192 bash[37887]: t=2023-07-10T17:01:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2023-07-10T17:01:03.784 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:03 smithi192 bash[37887]: t=2023-07-10T17:01:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2023-07-10T17:01:04.051 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:03 smithi192 bash[37887]: t=2023-07-10T17:01:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2023-07-10T17:01:04.052 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:03 smithi192 bash[37887]: t=2023-07-10T17:01:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2023-07-10T17:01:04.052 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:03 smithi192 bash[37887]: t=2023-07-10T17:01:03+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2023-07-10T17:01:04.362 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:04 smithi192 bash[37887]: t=2023-07-10T17:01:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2023-07-10T17:01:04.362 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:04 smithi192 bash[37887]: t=2023-07-10T17:01:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2023-07-10T17:01:04.362 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:04 smithi192 bash[37887]: t=2023-07-10T17:01:04+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2023-07-10T17:01:04.363 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:04 smithi192 bash[37887]: t=2023-07-10T17:01:04+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2023-07-10T17:01:04.363 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:04 smithi192 bash[37887]: t=2023-07-10T17:01:04+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2023-07-10T17:01:04.363 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:04 smithi192 bash[37887]: t=2023-07-10T17:01:04+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2023-07-10T17:01:04.621 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:04 smithi192 bash[23730]: cluster 2023-07-10T17:01:03.440926+0000 mgr.y (mgr.14144) 233 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:04.622 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:04 smithi192 bash[37887]: t=2023-07-10T17:01:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2023-07-10T17:01:04.622 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:04 smithi192 bash[37887]: t=2023-07-10T17:01:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2023-07-10T17:01:04.622 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:04 smithi192 bash[37887]: t=2023-07-10T17:01:04+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2023-07-10T17:01:04.850 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:04 smithi116 bash[23740]: cluster 2023-07-10T17:01:03.440926+0000 mgr.y (mgr.14144) 233 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:04.850 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:04 smithi116 bash[28351]: cluster 2023-07-10T17:01:03.440926+0000 mgr.y (mgr.14144) 233 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:04.885 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:04 smithi192 bash[37887]: t=2023-07-10T17:01:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2023-07-10T17:01:04.886 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:04 smithi192 bash[37887]: t=2023-07-10T17:01:04+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2023-07-10T17:01:04.886 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:04 smithi192 bash[37887]: t=2023-07-10T17:01:04+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2023-07-10T17:01:05.166 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:04 smithi192 bash[37887]: t=2023-07-10T17:01:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2023-07-10T17:01:05.166 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:04 smithi192 bash[37887]: t=2023-07-10T17:01:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2023-07-10T17:01:05.167 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:05 smithi192 bash[37887]: t=2023-07-10T17:01:05+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2023-07-10T17:01:05.167 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:05 smithi192 bash[37887]: t=2023-07-10T17:01:05+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2023-07-10T17:01:05.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:05 smithi192 bash[37887]: t=2023-07-10T17:01:05+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2023-07-10T17:01:05.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:05 smithi192 bash[37887]: t=2023-07-10T17:01:05+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2023-07-10T17:01:05.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:05 smithi192 bash[37887]: t=2023-07-10T17:01:05+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2023-07-10T17:01:05.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:05 smithi192 bash[37887]: t=2023-07-10T17:01:05+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2023-07-10T17:01:05.763 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:05 smithi192 bash[37887]: t=2023-07-10T17:01:05+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2023-07-10T17:01:05.763 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:05 smithi192 bash[37887]: t=2023-07-10T17:01:05+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2023-07-10T17:01:05.763 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:05 smithi192 bash[37887]: t=2023-07-10T17:01:05+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2023-07-10T17:01:05.763 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:05 smithi192 bash[37887]: t=2023-07-10T17:01:05+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2023-07-10T17:01:05.763 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:05 smithi192 bash[37887]: t=2023-07-10T17:01:05+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2023-07-10T17:01:05.763 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:05 smithi192 bash[37887]: t=2023-07-10T17:01:05+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2023-07-10T17:01:05.764 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:05 smithi192 bash[37887]: t=2023-07-10T17:01:05+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2023-07-10T17:01:06.081 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:05 smithi192 bash[37887]: t=2023-07-10T17:01:05+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2023-07-10T17:01:06.082 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:05 smithi192 bash[37887]: t=2023-07-10T17:01:05+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2023-07-10T17:01:06.082 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:05 smithi192 bash[37887]: t=2023-07-10T17:01:05+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2023-07-10T17:01:06.350 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:06 smithi192 bash[37887]: t=2023-07-10T17:01:06+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2023-07-10T17:01:06.350 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:06 smithi192 bash[37887]: t=2023-07-10T17:01:06+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2023-07-10T17:01:06.350 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:06 smithi192 bash[37887]: t=2023-07-10T17:01:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2023-07-10T17:01:06.350 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:06 smithi192 bash[37887]: t=2023-07-10T17:01:06+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2023-07-10T17:01:06.601 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:06 smithi192 bash[23730]: cluster 2023-07-10T17:01:05.441569+0000 mgr.y (mgr.14144) 234 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:06.601 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:06 smithi192 bash[37887]: t=2023-07-10T17:01:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2023-07-10T17:01:06.601 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:06 smithi192 bash[37887]: t=2023-07-10T17:01:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2023-07-10T17:01:06.602 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:06 smithi192 bash[37887]: t=2023-07-10T17:01:06+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2023-07-10T17:01:06.602 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:06 smithi192 bash[37887]: t=2023-07-10T17:01:06+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2023-07-10T17:01:06.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:06 smithi116 bash[23740]: cluster 2023-07-10T17:01:05.441569+0000 mgr.y (mgr.14144) 234 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:06.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:06 smithi116 bash[28351]: cluster 2023-07-10T17:01:05.441569+0000 mgr.y (mgr.14144) 234 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:06.885 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:06 smithi192 bash[37887]: t=2023-07-10T17:01:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2023-07-10T17:01:06.886 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:06 smithi192 bash[37887]: t=2023-07-10T17:01:06+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2023-07-10T17:01:06.886 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:06 smithi192 bash[37887]: t=2023-07-10T17:01:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2023-07-10T17:01:07.188 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:06 smithi192 bash[37887]: t=2023-07-10T17:01:06+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2023-07-10T17:01:07.188 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:06 smithi192 bash[37887]: t=2023-07-10T17:01:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2023-07-10T17:01:07.188 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:07 smithi192 bash[37887]: t=2023-07-10T17:01:07+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2023-07-10T17:01:07.188 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:07 smithi192 bash[37887]: t=2023-07-10T17:01:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2023-07-10T17:01:07.464 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:07 smithi192 bash[37887]: t=2023-07-10T17:01:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2023-07-10T17:01:07.465 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:07 smithi192 bash[37887]: t=2023-07-10T17:01:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2023-07-10T17:01:07.465 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:07 smithi192 bash[37887]: t=2023-07-10T17:01:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2023-07-10T17:01:07.465 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:07 smithi192 bash[37887]: t=2023-07-10T17:01:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2023-07-10T17:01:07.724 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:07 smithi192 bash[37887]: t=2023-07-10T17:01:07+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2023-07-10T17:01:07.725 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:07 smithi192 bash[37887]: t=2023-07-10T17:01:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2023-07-10T17:01:07.725 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:07 smithi192 bash[37887]: t=2023-07-10T17:01:07+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2023-07-10T17:01:07.725 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:07 smithi192 bash[37887]: t=2023-07-10T17:01:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2023-07-10T17:01:07.725 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:07 smithi192 bash[37887]: t=2023-07-10T17:01:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2023-07-10T17:01:08.018 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:07 smithi192 bash[37887]: t=2023-07-10T17:01:07+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2023-07-10T17:01:08.018 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:07 smithi192 bash[37887]: t=2023-07-10T17:01:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2023-07-10T17:01:08.018 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Created default admin" logger=sqlstore user=admin 2023-07-10T17:01:08.386 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing HTTPServer" logger=server 2023-07-10T17:01:08.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing BackendPluginManager" logger=server 2023-07-10T17:01:08.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing PluginManager" logger=server 2023-07-10T17:01:08.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Starting plugin search" logger=plugins 2023-07-10T17:01:08.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Registering plugin" logger=plugins name="Pie Chart" 2023-07-10T17:01:08.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Registering plugin" logger=plugins name="Status Panel" 2023-07-10T17:01:08.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing HooksService" logger=server 2023-07-10T17:01:08.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing OSSLicensingService" logger=server 2023-07-10T17:01:08.388 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing InternalMetricsService" logger=server 2023-07-10T17:01:08.388 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing RemoteCache" logger=server 2023-07-10T17:01:08.388 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing RenderingService" logger=server 2023-07-10T17:01:08.388 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing AlertEngine" logger=server 2023-07-10T17:01:08.388 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing QuotaService" logger=server 2023-07-10T17:01:08.388 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing ServerLockService" logger=server 2023-07-10T17:01:08.388 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2023-07-10T17:01:08.389 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server 2023-07-10T17:01:08.389 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing LoginService" logger=server 2023-07-10T17:01:08.389 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing SearchService" logger=server 2023-07-10T17:01:08.389 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing TracingService" logger=server 2023-07-10T17:01:08.389 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2023-07-10T17:01:08.389 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing CleanUpService" logger=server 2023-07-10T17:01:08.389 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing NotificationService" logger=server 2023-07-10T17:01:08.389 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2023-07-10T17:01:08.390 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2023-07-10T17:01:08.390 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 2023-07-10T17:01:08.390 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS 2023-07-10T17:01:08.390 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS 2023-07-10T17:01:08.390 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="Initializing Stream Manager" 2023-07-10T17:01:08.390 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[37887]: t=2023-07-10T17:01:08+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2023-07-10T17:01:08.786 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:08 smithi116 bash[23740]: cluster 2023-07-10T17:01:07.442135+0000 mgr.y (mgr.14144) 235 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:08.786 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:08 smithi116 bash[28351]: cluster 2023-07-10T17:01:07.442135+0000 mgr.y (mgr.14144) 235 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:08.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:08 smithi192 bash[23730]: cluster 2023-07-10T17:01:07.442135+0000 mgr.y (mgr.14144) 235 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:09.937 INFO:teuthology.orchestra.run.smithi116.stdout:111669149734 2023-07-10T17:01:09.938 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph osd last-stat-seq osd.3 2023-07-10T17:01:10.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:10 smithi192 bash[23730]: cluster 2023-07-10T17:01:09.442750+0000 mgr.y (mgr.14144) 236 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:10.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:10 smithi116 bash[28351]: cluster 2023-07-10T17:01:09.442750+0000 mgr.y (mgr.14144) 236 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:10.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:10 smithi116 bash[23740]: cluster 2023-07-10T17:01:09.442750+0000 mgr.y (mgr.14144) 236 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:11.465 INFO:teuthology.orchestra.run.smithi116.stdout:34359738422 2023-07-10T17:01:11.465 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph osd last-stat-seq osd.0 2023-07-10T17:01:12.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:12 smithi192 bash[23730]: cluster 2023-07-10T17:01:11.443493+0000 mgr.y (mgr.14144) 237 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:12.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:12 smithi116 bash[23740]: cluster 2023-07-10T17:01:11.443493+0000 mgr.y (mgr.14144) 237 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:12.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:12 smithi116 bash[28351]: cluster 2023-07-10T17:01:11.443493+0000 mgr.y (mgr.14144) 237 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:13.445 INFO:teuthology.orchestra.run.smithi116.stdout:55834574899 2023-07-10T17:01:13.446 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph osd last-stat-seq osd.1 2023-07-10T17:01:13.670 INFO:teuthology.orchestra.run.smithi116.stdout:184683593754 2023-07-10T17:01:13.671 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph osd last-stat-seq osd.6 2023-07-10T17:01:14.706 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:14 smithi116 bash[23740]: cluster 2023-07-10T17:01:13.444215+0000 mgr.y (mgr.14144) 238 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:14.707 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:14 smithi116 bash[28351]: cluster 2023-07-10T17:01:13.444215+0000 mgr.y (mgr.14144) 238 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:14.865 INFO:teuthology.orchestra.run.smithi116.stdout:133143986212 2023-07-10T17:01:14.866 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph osd last-stat-seq osd.4 2023-07-10T17:01:14.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:14 smithi192 bash[23730]: cluster 2023-07-10T17:01:13.444215+0000 mgr.y (mgr.14144) 238 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:15.034 INFO:teuthology.orchestra.run.smithi116.stdout:210453397525 2023-07-10T17:01:15.034 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph osd last-stat-seq osd.7 2023-07-10T17:01:16.213 INFO:teuthology.orchestra.run.smithi116.stdout:77309411374 2023-07-10T17:01:16.214 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph osd last-stat-seq osd.2 2023-07-10T17:01:16.498 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:16 smithi116 bash[28351]: cluster 2023-07-10T17:01:15.444896+0000 mgr.y (mgr.14144) 239 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:16.498 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:16 smithi116 bash[23740]: cluster 2023-07-10T17:01:15.444896+0000 mgr.y (mgr.14144) 239 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:16.549 INFO:teuthology.orchestra.run.smithi116.stdout:158913789984 2023-07-10T17:01:16.549 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph osd last-stat-seq osd.5 2023-07-10T17:01:16.762 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:01:16 smithi116 systemd[1]: Stopping Ceph alertmanager.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:01:16.763 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:01:16 smithi116 bash[41991]: level=info ts=2023-07-10T17:01:16.583Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2023-07-10T17:01:16.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:16 smithi192 bash[23730]: cluster 2023-07-10T17:01:15.444896+0000 mgr.y (mgr.14144) 239 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:16.974 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:01:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:01:18.142 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:01:17 smithi116 docker[44794]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-alertmanager.a 2023-07-10T17:01:18.142 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:01:17 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@alertmanager.a.service: Succeeded. 2023-07-10T17:01:18.142 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:01:17 smithi116 systemd[1]: Stopped Ceph alertmanager.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:01:18.142 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:01:17 smithi116 systemd[1]: Started Ceph alertmanager.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:01:18.289 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:01:18.290 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:01:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:01:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:01:18.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:18 smithi192 bash[23730]: cluster 2023-07-10T17:01:17.445473+0000 mgr.y (mgr.14144) 240 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:18.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:18 smithi192 bash[23730]: audit 2023-07-10T17:01:17.873756+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:18.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:18 smithi116 bash[28351]: cluster 2023-07-10T17:01:17.445473+0000 mgr.y (mgr.14144) 240 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:18.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:18 smithi116 bash[28351]: audit 2023-07-10T17:01:17.873756+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:18.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:18 smithi116 bash[23740]: cluster 2023-07-10T17:01:17.445473+0000 mgr.y (mgr.14144) 240 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:18.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:18 smithi116 bash[23740]: audit 2023-07-10T17:01:17.873756+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:19.315 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:01:19.316 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:01:19.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:19 smithi192 bash[23730]: cephadm 2023-07-10T17:01:17.876019+0000 mgr.y (mgr.14144) 241 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-10T17:01:19.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:19 smithi192 bash[23730]: cephadm 2023-07-10T17:01:17.882222+0000 mgr.y (mgr.14144) 242 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi192 2023-07-10T17:01:19.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:19 smithi116 bash[23740]: cephadm 2023-07-10T17:01:17.876019+0000 mgr.y (mgr.14144) 241 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-10T17:01:19.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:19 smithi116 bash[23740]: cephadm 2023-07-10T17:01:17.882222+0000 mgr.y (mgr.14144) 242 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi192 2023-07-10T17:01:19.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:19 smithi116 bash[28351]: cephadm 2023-07-10T17:01:17.876019+0000 mgr.y (mgr.14144) 241 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-10T17:01:19.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:19 smithi116 bash[28351]: cephadm 2023-07-10T17:01:17.882222+0000 mgr.y (mgr.14144) 242 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi192 2023-07-10T17:01:20.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:20 smithi116 bash[23740]: cluster 2023-07-10T17:01:19.446066+0000 mgr.y (mgr.14144) 243 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:20.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:20 smithi116 bash[23740]: audit 2023-07-10T17:01:19.570388+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:01:20.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:20 smithi116 bash[23740]: audit 2023-07-10T17:01:19.574379+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:01:20.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:20 smithi116 bash[28351]: cluster 2023-07-10T17:01:19.446066+0000 mgr.y (mgr.14144) 243 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:20.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:20 smithi116 bash[28351]: audit 2023-07-10T17:01:19.570388+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:01:20.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:20 smithi116 bash[28351]: audit 2023-07-10T17:01:19.574379+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:01:20.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:20 smithi192 bash[23730]: cluster 2023-07-10T17:01:19.446066+0000 mgr.y (mgr.14144) 243 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:20.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:20 smithi192 bash[23730]: audit 2023-07-10T17:01:19.570388+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:01:20.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:20 smithi192 bash[23730]: audit 2023-07-10T17:01:19.574379+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:01:22.250 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:21 smithi192 systemd[1]: Stopping Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:01:22.250 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:22 smithi192 bash[36820]: level=warn ts=2023-07-10T17:01:22.033Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..." 2023-07-10T17:01:22.250 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:22 smithi192 bash[36820]: level=info ts=2023-07-10T17:01:22.034Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-10T17:01:22.251 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:22 smithi192 bash[36820]: level=info ts=2023-07-10T17:01:22.034Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-10T17:01:22.251 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:22 smithi192 bash[36820]: level=info ts=2023-07-10T17:01:22.034Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-10T17:01:22.251 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:22 smithi192 bash[36820]: level=info ts=2023-07-10T17:01:22.034Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-10T17:01:22.251 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:22 smithi192 bash[36820]: level=info ts=2023-07-10T17:01:22.034Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-10T17:01:22.251 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:22 smithi192 bash[36820]: level=info ts=2023-07-10T17:01:22.034Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-10T17:01:22.252 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:22 smithi192 bash[36820]: level=info ts=2023-07-10T17:01:22.034Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-10T17:01:22.252 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:22 smithi192 bash[36820]: level=info ts=2023-07-10T17:01:22.034Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-10T17:01:22.252 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:22 smithi192 bash[36820]: level=info ts=2023-07-10T17:01:22.034Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-10T17:01:22.252 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:22 smithi192 bash[36820]: level=info ts=2023-07-10T17:01:22.034Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-10T17:01:22.252 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:22 smithi192 bash[36820]: level=info ts=2023-07-10T17:01:22.035Z caller=main.go:761 msg="See you next time!" 2023-07-10T17:01:22.538 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:22 smithi116 bash[23740]: cluster 2023-07-10T17:01:21.446690+0000 mgr.y (mgr.14144) 244 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:22.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:22 smithi192 bash[23730]: cluster 2023-07-10T17:01:21.446690+0000 mgr.y (mgr.14144) 244 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:22.669 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:22 smithi116 bash[28351]: cluster 2023-07-10T17:01:21.446690+0000 mgr.y (mgr.14144) 244 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:23.135 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:22 smithi192 docker[38483]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-prometheus.a 2023-07-10T17:01:23.136 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:22 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Succeeded. 2023-07-10T17:01:23.136 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:22 smithi192 systemd[1]: Stopped Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:01:23.137 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:22 smithi192 systemd[1]: Started Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:01:24.081 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.739534+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:24.082 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.746270+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:01:24.082 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.747007+0000 mgr.y (mgr.14144) 245 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:01:24.082 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.749185+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi116:9093"}]: dispatch 2023-07-10T17:01:24.082 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.749812+0000 mgr.y (mgr.14144) 246 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi116:9093"}]: dispatch 2023-07-10T17:01:24.082 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.758592+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:24.082 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.766945+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:01:24.083 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.767602+0000 mgr.y (mgr.14144) 247 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:01:24.083 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.775032+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192:9095"}]: dispatch 2023-07-10T17:01:24.083 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.775634+0000 mgr.y (mgr.14144) 248 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192:9095"}]: dispatch 2023-07-10T17:01:24.083 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.784823+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:24.083 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.790584+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:01:24.083 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.791296+0000 mgr.y (mgr.14144) 249 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:01:24.083 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.801663+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192:3000"}]: dispatch 2023-07-10T17:01:24.083 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.802225+0000 mgr.y (mgr.14144) 250 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192:3000"}]: dispatch 2023-07-10T17:01:24.084 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.811600+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:24.084 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: audit 2023-07-10T17:01:22.816464+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:01:24.084 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[28351]: cluster 2023-07-10T17:01:23.447164+0000 mgr.y (mgr.14144) 251 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:24.084 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.739534+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:24.085 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.746270+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:01:24.085 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.747007+0000 mgr.y (mgr.14144) 245 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:01:24.085 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.749185+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi116:9093"}]: dispatch 2023-07-10T17:01:24.085 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.749812+0000 mgr.y (mgr.14144) 246 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi116:9093"}]: dispatch 2023-07-10T17:01:24.085 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.758592+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:24.085 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.766945+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:01:24.085 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.767602+0000 mgr.y (mgr.14144) 247 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:01:24.086 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.775032+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192:9095"}]: dispatch 2023-07-10T17:01:24.086 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.775634+0000 mgr.y (mgr.14144) 248 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192:9095"}]: dispatch 2023-07-10T17:01:24.086 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.784823+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:24.086 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.790584+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:01:24.086 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.791296+0000 mgr.y (mgr.14144) 249 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:01:24.086 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.801663+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192:3000"}]: dispatch 2023-07-10T17:01:24.086 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.802225+0000 mgr.y (mgr.14144) 250 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192:3000"}]: dispatch 2023-07-10T17:01:24.086 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.811600+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:24.087 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: audit 2023-07-10T17:01:22.816464+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:01:24.087 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:23 smithi116 bash[23740]: cluster 2023-07-10T17:01:23.447164+0000 mgr.y (mgr.14144) 251 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:24.136 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.843Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-10T17:01:24.136 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.843Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-10T17:01:24.137 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.843Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-10T17:01:24.137 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.843Z caller=main.go:339 host_details="(Linux 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 smithi192 (none))" 2023-07-10T17:01:24.137 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.843Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-10T17:01:24.138 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.843Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-10T17:01:24.138 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.845Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-10T17:01:24.138 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.845Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-10T17:01:24.138 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.850Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-10T17:01:24.139 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.851Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2023-07-10T17:01:24.139 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.852Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2023-07-10T17:01:24.139 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.852Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=1.510492ms 2023-07-10T17:01:24.139 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.854Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-10T17:01:24.140 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.854Z caller=main.go:695 msg="TSDB started" 2023-07-10T17:01:24.140 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.854Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-10T17:01:24.140 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.855Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-10T17:01:24.141 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[38545]: level=info ts=2023-07-10T17:01:23.855Z caller=main.go:646 msg="Server is ready to receive web requests." 2023-07-10T17:01:24.141 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.739534+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:24.141 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.746270+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:01:24.142 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.747007+0000 mgr.y (mgr.14144) 245 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:01:24.142 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.749185+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi116:9093"}]: dispatch 2023-07-10T17:01:24.142 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.749812+0000 mgr.y (mgr.14144) 246 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi116:9093"}]: dispatch 2023-07-10T17:01:24.143 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.758592+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:24.143 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.766945+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:01:24.143 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.767602+0000 mgr.y (mgr.14144) 247 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:01:24.143 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.775032+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192:9095"}]: dispatch 2023-07-10T17:01:24.144 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.775634+0000 mgr.y (mgr.14144) 248 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192:9095"}]: dispatch 2023-07-10T17:01:24.144 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.784823+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:24.144 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.790584+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:01:24.145 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.791296+0000 mgr.y (mgr.14144) 249 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:01:24.145 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.801663+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192:3000"}]: dispatch 2023-07-10T17:01:24.145 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.802225+0000 mgr.y (mgr.14144) 250 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192:3000"}]: dispatch 2023-07-10T17:01:24.145 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.811600+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:24.146 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: audit 2023-07-10T17:01:22.816464+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:01:24.146 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:23 smithi192 bash[23730]: cluster 2023-07-10T17:01:23.447164+0000 mgr.y (mgr.14144) 251 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:24.908 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:01:24 smithi116 bash[44891]: level=info ts=2023-07-10T17:01:24.601Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2023-07-10T17:01:24.908 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:01:24 smithi116 bash[44891]: level=info ts=2023-07-10T17:01:24.601Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2023-07-10T17:01:24.908 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:01:24 smithi116 bash[44891]: level=info ts=2023-07-10T17:01:24.603Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.116 port=9094 2023-07-10T17:01:24.909 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:01:24 smithi116 bash[44891]: level=info ts=2023-07-10T17:01:24.605Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2023-07-10T17:01:24.909 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:01:24 smithi116 bash[44891]: level=info ts=2023-07-10T17:01:24.644Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-10T17:01:24.909 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:01:24 smithi116 bash[44891]: level=info ts=2023-07-10T17:01:24.645Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-10T17:01:24.909 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:01:24 smithi116 bash[44891]: level=info ts=2023-07-10T17:01:24.650Z caller=main.go:497 msg=Listening address=:9093 2023-07-10T17:01:26.592 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:26 smithi116 bash[23740]: cluster 2023-07-10T17:01:25.447560+0000 mgr.y (mgr.14144) 252 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:26.592 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:26 smithi116 bash[28351]: cluster 2023-07-10T17:01:25.447560+0000 mgr.y (mgr.14144) 252 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:26.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:26 smithi192 bash[23730]: cluster 2023-07-10T17:01:25.447560+0000 mgr.y (mgr.14144) 252 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:26.892 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:01:26 smithi116 bash[44891]: level=info ts=2023-07-10T17:01:26.605Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000184227s 2023-07-10T17:01:27.074 INFO:teuthology.orchestra.run.smithi116.stdout:111669149738 2023-07-10T17:01:27.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:27 smithi116 bash[23740]: audit 2023-07-10T17:01:26.684330+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:27.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:27 smithi116 bash[23740]: audit 2023-07-10T17:01:27.072427+0000 mon.a (mon.0) 511 : audit [DBG] from='client.? 172.21.15.116:0/2462133667' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-10T17:01:27.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:27 smithi116 bash[28351]: audit 2023-07-10T17:01:26.684330+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:27.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:27 smithi116 bash[28351]: audit 2023-07-10T17:01:27.072427+0000 mon.a (mon.0) 511 : audit [DBG] from='client.? 172.21.15.116:0/2462133667' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-10T17:01:28.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:27 smithi192 bash[23730]: audit 2023-07-10T17:01:26.684330+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:28.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:27 smithi192 bash[23730]: audit 2023-07-10T17:01:27.072427+0000 mon.a (mon.0) 511 : audit [DBG] from='client.? 172.21.15.116:0/2462133667' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-10T17:01:28.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:28 smithi116 bash[28351]: cluster 2023-07-10T17:01:27.448111+0000 mgr.y (mgr.14144) 253 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:28.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:28 smithi116 bash[23740]: cluster 2023-07-10T17:01:27.448111+0000 mgr.y (mgr.14144) 253 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:28.932 INFO:teuthology.orchestra.run.smithi116.stdout:34359738426 2023-07-10T17:01:29.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:28 smithi192 bash[23730]: cluster 2023-07-10T17:01:27.448111+0000 mgr.y (mgr.14144) 253 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:30.039 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:29 smithi116 bash[28351]: audit 2023-07-10T17:01:28.930218+0000 mon.a (mon.0) 512 : audit [DBG] from='client.? 172.21.15.116:0/1505484216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-10T17:01:30.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:29 smithi116 bash[23740]: audit 2023-07-10T17:01:28.930218+0000 mon.a (mon.0) 512 : audit [DBG] from='client.? 172.21.15.116:0/1505484216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-10T17:01:30.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:29 smithi192 bash[23730]: audit 2023-07-10T17:01:28.930218+0000 mon.a (mon.0) 512 : audit [DBG] from='client.? 172.21.15.116:0/1505484216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-10T17:01:31.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:30 smithi192 bash[23730]: cluster 2023-07-10T17:01:29.448800+0000 mgr.y (mgr.14144) 254 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:31.142 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:30 smithi116 bash[23740]: cluster 2023-07-10T17:01:29.448800+0000 mgr.y (mgr.14144) 254 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:31.142 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:30 smithi116 bash[28351]: cluster 2023-07-10T17:01:29.448800+0000 mgr.y (mgr.14144) 254 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:31.997 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149734 got 111669149738 for osd.3 2023-07-10T17:01:31.997 DEBUG:teuthology.parallel:result is None 2023-07-10T17:01:32.260 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738422 got 34359738426 for osd.0 2023-07-10T17:01:32.261 DEBUG:teuthology.parallel:result is None 2023-07-10T17:01:33.032 INFO:teuthology.orchestra.run.smithi116.stdout:55834574903 2023-07-10T17:01:33.117 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:32 smithi116 bash[28351]: cluster 2023-07-10T17:01:31.449486+0000 mgr.y (mgr.14144) 255 : cluster [DBG] pgmap v227: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:33.117 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:32 smithi116 bash[23740]: cluster 2023-07-10T17:01:31.449486+0000 mgr.y (mgr.14144) 255 : cluster [DBG] pgmap v227: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:33.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:32 smithi192 bash[23730]: cluster 2023-07-10T17:01:31.449486+0000 mgr.y (mgr.14144) 255 : cluster [DBG] pgmap v227: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:33.239 INFO:teuthology.orchestra.run.smithi116.stdout:210453397529 2023-07-10T17:01:34.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:33 smithi192 bash[23730]: audit 2023-07-10T17:01:33.029932+0000 mon.a (mon.0) 513 : audit [DBG] from='client.? 172.21.15.116:0/1431756015' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-10T17:01:34.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:33 smithi192 bash[23730]: audit 2023-07-10T17:01:33.237403+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.116:0/799344163' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-10T17:01:34.137 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:33 smithi116 bash[28351]: audit 2023-07-10T17:01:33.029932+0000 mon.a (mon.0) 513 : audit [DBG] from='client.? 172.21.15.116:0/1431756015' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-10T17:01:34.137 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:33 smithi116 bash[28351]: audit 2023-07-10T17:01:33.237403+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.116:0/799344163' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-10T17:01:34.137 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:33 smithi116 bash[23740]: audit 2023-07-10T17:01:33.029932+0000 mon.a (mon.0) 513 : audit [DBG] from='client.? 172.21.15.116:0/1431756015' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-10T17:01:34.138 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:33 smithi116 bash[23740]: audit 2023-07-10T17:01:33.237403+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.116:0/799344163' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-10T17:01:34.892 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:01:34 smithi116 bash[44891]: level=info ts=2023-07-10T17:01:34.606Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.001126287s 2023-07-10T17:01:34.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:34 smithi116 bash[23740]: cluster 2023-07-10T17:01:33.449887+0000 mgr.y (mgr.14144) 256 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:34.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:34 smithi116 bash[23740]: audit 2023-07-10T17:01:33.984837+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:34.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:34 smithi116 bash[23740]: audit 2023-07-10T17:01:34.231599+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:34.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:34 smithi116 bash[23740]: audit 2023-07-10T17:01:34.237746+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:01:34.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:34 smithi116 bash[28351]: cluster 2023-07-10T17:01:33.449887+0000 mgr.y (mgr.14144) 256 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:34.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:34 smithi116 bash[28351]: audit 2023-07-10T17:01:33.984837+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:34.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:34 smithi116 bash[28351]: audit 2023-07-10T17:01:34.231599+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:34.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:34 smithi116 bash[28351]: audit 2023-07-10T17:01:34.237746+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:01:35.004 INFO:teuthology.orchestra.run.smithi116.stdout:133143986217 2023-07-10T17:01:35.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:34 smithi192 bash[23730]: cluster 2023-07-10T17:01:33.449887+0000 mgr.y (mgr.14144) 256 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:35.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:34 smithi192 bash[23730]: audit 2023-07-10T17:01:33.984837+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:35.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:34 smithi192 bash[23730]: audit 2023-07-10T17:01:34.231599+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:35.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:34 smithi192 bash[23730]: audit 2023-07-10T17:01:34.237746+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:01:35.303 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574899 got 55834574903 for osd.1 2023-07-10T17:01:35.304 DEBUG:teuthology.parallel:result is None 2023-07-10T17:01:35.987 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:35 smithi116 bash[28351]: audit 2023-07-10T17:01:35.000467+0000 mon.c (mon.1) 21 : audit [DBG] from='client.? 172.21.15.116:0/1445079815' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-10T17:01:35.988 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:35 smithi116 bash[23740]: audit 2023-07-10T17:01:35.000467+0000 mon.c (mon.1) 21 : audit [DBG] from='client.? 172.21.15.116:0/1445079815' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-10T17:01:36.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:35 smithi192 bash[23730]: audit 2023-07-10T17:01:35.000467+0000 mon.c (mon.1) 21 : audit [DBG] from='client.? 172.21.15.116:0/1445079815' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-10T17:01:36.258 INFO:teuthology.orchestra.run.smithi116.stdout:184683593758 2023-07-10T17:01:36.756 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397525 got 210453397529 for osd.7 2023-07-10T17:01:36.757 DEBUG:teuthology.parallel:result is None 2023-07-10T17:01:37.072 INFO:teuthology.orchestra.run.smithi116.stdout:158913789988 2023-07-10T17:01:37.077 INFO:teuthology.orchestra.run.smithi116.stdout:77309411378 2023-07-10T17:01:37.118 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:36 smithi116 bash[28351]: cluster 2023-07-10T17:01:35.450356+0000 mgr.y (mgr.14144) 257 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:37.118 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:36 smithi116 bash[28351]: audit 2023-07-10T17:01:36.255220+0000 mon.c (mon.1) 22 : audit [DBG] from='client.? 172.21.15.116:0/2386793899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-10T17:01:37.119 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:36 smithi116 bash[23740]: cluster 2023-07-10T17:01:35.450356+0000 mgr.y (mgr.14144) 257 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:37.119 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:36 smithi116 bash[23740]: audit 2023-07-10T17:01:36.255220+0000 mon.c (mon.1) 22 : audit [DBG] from='client.? 172.21.15.116:0/2386793899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-10T17:01:37.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:36 smithi192 bash[23730]: cluster 2023-07-10T17:01:35.450356+0000 mgr.y (mgr.14144) 257 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:37.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:36 smithi192 bash[23730]: audit 2023-07-10T17:01:36.255220+0000 mon.c (mon.1) 22 : audit [DBG] from='client.? 172.21.15.116:0/2386793899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-10T17:01:37.572 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986212 got 133143986217 for osd.4 2023-07-10T17:01:37.573 DEBUG:teuthology.parallel:result is None 2023-07-10T17:01:37.827 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:37 smithi116 bash[23740]: audit 2023-07-10T17:01:37.069447+0000 mon.c (mon.1) 23 : audit [DBG] from='client.? 172.21.15.116:0/201316998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-10T17:01:37.827 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:37 smithi116 bash[23740]: audit 2023-07-10T17:01:37.074880+0000 mon.c (mon.1) 24 : audit [DBG] from='client.? 172.21.15.116:0/2067708881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-10T17:01:37.827 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:37 smithi116 bash[23740]: cluster 2023-07-10T17:01:37.450975+0000 mgr.y (mgr.14144) 258 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:37.827 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:37 smithi116 bash[28351]: audit 2023-07-10T17:01:37.069447+0000 mon.c (mon.1) 23 : audit [DBG] from='client.? 172.21.15.116:0/201316998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-10T17:01:37.828 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:37 smithi116 bash[28351]: audit 2023-07-10T17:01:37.074880+0000 mon.c (mon.1) 24 : audit [DBG] from='client.? 172.21.15.116:0/2067708881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-10T17:01:37.828 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:37 smithi116 bash[28351]: cluster 2023-07-10T17:01:37.450975+0000 mgr.y (mgr.14144) 258 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:38.061 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593754 got 184683593758 for osd.6 2023-07-10T17:01:38.062 DEBUG:teuthology.parallel:result is None 2023-07-10T17:01:38.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:37 smithi192 bash[23730]: audit 2023-07-10T17:01:37.069447+0000 mon.c (mon.1) 23 : audit [DBG] from='client.? 172.21.15.116:0/201316998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-10T17:01:38.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:37 smithi192 bash[23730]: audit 2023-07-10T17:01:37.074880+0000 mon.c (mon.1) 24 : audit [DBG] from='client.? 172.21.15.116:0/2067708881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-10T17:01:38.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:37 smithi192 bash[23730]: cluster 2023-07-10T17:01:37.450975+0000 mgr.y (mgr.14144) 258 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:38.730 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411374 got 77309411378 for osd.2 2023-07-10T17:01:38.731 DEBUG:teuthology.parallel:result is None 2023-07-10T17:01:38.736 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789984 got 158913789988 for osd.5 2023-07-10T17:01:38.736 DEBUG:teuthology.parallel:result is None 2023-07-10T17:01:38.736 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-07-10T17:01:38.737 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph pg dump --format=json 2023-07-10T17:01:39.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:39 smithi192 bash[23730]: audit 2023-07-10T17:01:38.120873+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:39.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:39 smithi116 bash[23740]: audit 2023-07-10T17:01:38.120873+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:39.392 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:39 smithi116 bash[28351]: audit 2023-07-10T17:01:38.120873+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:40.385 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:40 smithi192 bash[23730]: cluster 2023-07-10T17:01:39.451593+0000 mgr.y (mgr.14144) 259 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:40.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:40 smithi116 bash[23740]: cluster 2023-07-10T17:01:39.451593+0000 mgr.y (mgr.14144) 259 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:40.392 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:40 smithi116 bash[28351]: cluster 2023-07-10T17:01:39.451593+0000 mgr.y (mgr.14144) 259 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:01:41.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:41 smithi192 bash[23730]: audit 2023-07-10T17:01:40.145037+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:41.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:41 smithi192 bash[23730]: audit 2023-07-10T17:01:40.151278+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:41.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:41 smithi116 bash[28351]: audit 2023-07-10T17:01:40.145037+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:41.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:41 smithi116 bash[28351]: audit 2023-07-10T17:01:40.151278+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:41.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:41 smithi116 bash[23740]: audit 2023-07-10T17:01:40.145037+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:41.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:41 smithi116 bash[23740]: audit 2023-07-10T17:01:40.151278+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:01:42.303 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:42 smithi116 bash[23740]: cluster 2023-07-10T17:01:41.452174+0000 mgr.y (mgr.14144) 260 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:42.304 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:42 smithi116 bash[28351]: cluster 2023-07-10T17:01:41.452174+0000 mgr.y (mgr.14144) 260 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:42.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:42 smithi192 bash[23730]: cluster 2023-07-10T17:01:41.452174+0000 mgr.y (mgr.14144) 260 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:44.263 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-07-10T17:01:44.266 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-07-10T17:01:44.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:44 smithi116 bash[23740]: cluster 2023-07-10T17:01:43.452630+0000 mgr.y (mgr.14144) 261 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:44.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:44 smithi116 bash[28351]: cluster 2023-07-10T17:01:43.452630+0000 mgr.y (mgr.14144) 261 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:44.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:44 smithi192 bash[23730]: cluster 2023-07-10T17:01:43.452630+0000 mgr.y (mgr.14144) 261 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:45.016 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2023-07-10T17:01:43.452419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":43256,"kb_used_data":3768,"kb_used_omap":0,"kb_used_meta":39424,"kb_avail":749917960,"statfs":{"total":767960285184,"available":767915991040,"internally_reserved":0,"allocated":3858432,"data_stored":1530234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40370176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"51","state":"active+clean","last_fresh":"2023-07-10T16:59:41.552040+0000","last_change":"2023-07-10T16:59:41.552040+0000","last_active":"2023-07-10T16:59:41.552040+0000","last_peered":"2023-07-10T16:59:41.552040+0000","last_clean":"2023-07-10T16:59:41.552040+0000","last_became_active":"2023-07-10T16:59:41.551241+0000","last_became_peered":"2023-07-10T16:59:41.551241+0000","last_unstale":"2023-07-10T16:59:41.552040+0000","last_undegraded":"2023-07-10T16:59:41.552040+0000","last_fullsized":"2023-07-10T16:59:41.552040+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-10T16:57:56.919074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-10T16:57:56.919074+0000","last_clean_scrub_stamp":"2023-07-10T16:57:56.919074+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397531,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5216,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93739936,"statfs":{"total":95995035648,"available":95989694464,"internally_reserved":0,"allocated":483328,"data_stored":191358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jul 10 17:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.306},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":1.2390000000000001}]},{"osd":1,"last update":"Mon Jul 10 17:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":1.0429999999999999}]},{"osd":2,"last update":"Mon Jul 10 17:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":1.361}]},{"osd":3,"last update":"Mon Jul 10 17:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":1.006},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Jul 10 17:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":1.2110000000000001}]},{"osd":5,"last update":"Mon Jul 10 17:00:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.7110000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.292}]},{"osd":6,"last update":"Mon Jul 10 17:00:41 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.143,"15min":1.143},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.155},{"interface":"front","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":1.276}]}]},{"osd":6,"up_from":43,"seq":184683593760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5280,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93739872,"statfs":{"total":95995035648,"available":95989628928,"internally_reserved":0,"allocated":483328,"data_stored":191358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jul 10 17:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.3400000000000001,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":1.113},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.94699999999999995,"15min":0.95199999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":0.95499999999999996}]},{"osd":1,"last update":"Mon Jul 10 17:01:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.016,"15min":1.014},"min":{"1min":0.64900000000000002,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.268,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0880000000000001,"15min":1.0920000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":1.234}]},{"osd":2,"last update":"Mon Jul 10 17:01:17 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.012,"15min":1.0109999999999999},"min":{"1min":0.73699999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.4019999999999999,"5min":1.552,"15min":1.552},"last":1.079},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1,"15min":1.0089999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":1.1759999999999999}]},{"osd":3,"last update":"Mon Jul 10 17:01:17 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0089999999999999,"15min":1.0009999999999999},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.222},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.121,"15min":1.131},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.248,"5min":1.635,"15min":1.635},"last":1.456}]},{"osd":4,"last update":"Mon Jul 10 17:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.98499999999999999,"15min":0.99299999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.089,"5min":1.109,"15min":1.1120000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":1.0049999999999999}]},{"osd":5,"last update":"Mon Jul 10 17:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.0509999999999999,"15min":1.0609999999999999},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.2709999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":1.046},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1779999999999999,"15min":1.1759999999999999},"min":{"1min":0.86799999999999999,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.5920000000000001,"5min":1.645,"15min":1.645},"last":1.2869999999999999}]},{"osd":7,"last update":"Mon Jul 10 17:00:42 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.214,"15min":1.214},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.363},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":1.3500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5532,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739620,"statfs":{"total":95995035648,"available":95989370880,"internally_reserved":0,"allocated":479232,"data_stored":191043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jul 10 17:01:24 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.67500000000000004,"15min":0.60699999999999998},"min":{"1min":0.49299999999999999,"5min":0.442,"15min":0.442},"max":{"1min":1.131,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.72799999999999998,"15min":0.66200000000000003},"min":{"1min":0.443,"5min":0.436,"15min":0.436},"max":{"1min":1.101,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.69799999999999995}]},{"osd":2,"last update":"Mon Jul 10 17:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.77900000000000003,"15min":0.73699999999999999},"min":{"1min":0.55600000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.80700000000000005,"15min":0.76700000000000002},"min":{"1min":0.52800000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.218,"5min":1.228,"15min":1.228},"last":0.95199999999999996}]},{"osd":3,"last update":"Mon Jul 10 17:01:14 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81299999999999994,"15min":0.80500000000000005},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.84799999999999998,"15min":0.83399999999999996},"min":{"1min":0.64400000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.732,"5min":1.732,"15min":1.732},"last":1.1910000000000001}]},{"osd":4,"last update":"Mon Jul 10 17:01:32 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.008,"15min":1.004},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.4199999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0409999999999999,"15min":1.0149999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.3879999999999999,"5min":2.1349999999999998,"15min":2.1349999999999998},"last":1.413}]},{"osd":5,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.212,"15min":1.23},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.5029999999999999,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.145,"5min":1.179,"15min":1.1850000000000001},"min":{"1min":0.90000000000000002,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.3819999999999999,"5min":1.431,"15min":1.431},"last":1.2290000000000001}]},{"osd":6,"last update":"Mon Jul 10 17:01:20 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.22,"15min":1.238},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.587,"5min":1.821,"15min":1.821},"last":1.524},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.254,"15min":1.268},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.474,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":1.355}]},{"osd":7,"last update":"Mon Jul 10 17:00:40 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.474},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":1.071,"5min":1.071,"15min":1.071},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.276}]}]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5596,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739556,"statfs":{"total":95995035648,"available":95989305344,"internally_reserved":0,"allocated":479232,"data_stored":191043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jul 10 17:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.69599999999999995,"15min":0.64000000000000001},"min":{"1min":0.61399999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1359999999999999,"5min":1.248,"15min":1.248},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.79000000000000004,"15min":0.74399999999999999},"min":{"1min":0.70299999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0249999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":1.109}]},{"osd":2,"last update":"Mon Jul 10 17:00:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.69999999999999996,"15min":0.65900000000000003},"min":{"1min":0.44900000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.0700000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":1.0660000000000001},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.73999999999999999,"15min":0.70699999999999996},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.93899999999999995}]},{"osd":3,"last update":"Mon Jul 10 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.748,"15min":0.72699999999999998},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.085,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.87},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.91900000000000004,"15min":0.88700000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.3129999999999999,"5min":1.377,"15min":1.377},"last":1.1679999999999999}]},{"osd":4,"last update":"Mon Jul 10 17:01:33 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0669999999999999,"15min":1.077},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.363,"5min":1.466,"15min":1.466},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0549999999999999,"15min":1.0389999999999999},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.651,"5min":1.651,"15min":1.651},"last":1.143}]},{"osd":5,"last update":"Mon Jul 10 17:00:59 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.157,"15min":1.155},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.109,"5min":1.095,"15min":1.093},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":1.095}]},{"osd":6,"last update":"Mon Jul 10 17:01:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2210000000000001,"15min":1.2230000000000001},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":1.5089999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.194},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.214,"15min":1.2170000000000001},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"max":{"1min":1.466,"5min":1.466,"15min":1.466},"last":1.3380000000000001}]},{"osd":7,"last update":"Mon Jul 10 17:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.25,"15min":1.256},"min":{"1min":0.88700000000000001,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.5669999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.365},{"interface":"front","average":{"1min":1.258,"5min":1.214,"15min":1.206},"min":{"1min":0.92000000000000004,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.454}]}]},{"osd":2,"up_from":18,"seq":77309411380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5476,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739676,"statfs":{"total":95995035648,"available":95989428224,"internally_reserved":0,"allocated":487424,"data_stored":191673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jul 10 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.66200000000000003,"15min":0.629},"min":{"1min":0.57099999999999995,"5min":0.496,"15min":0.496},"max":{"1min":1.1910000000000001,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84599999999999997,"15min":0.85199999999999998},"min":{"1min":0.61899999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.282,"5min":3.9569999999999999,"15min":3.9569999999999999},"last":0.89600000000000002}]},{"osd":1,"last update":"Mon Jul 10 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.83599999999999997,"15min":0.85699999999999998},"min":{"1min":0.58899999999999997,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1000000000000001,"5min":3.9900000000000002,"15min":3.9900000000000002},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.81999999999999995,"15min":0.77800000000000002},"min":{"1min":0.63800000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.622}]},{"osd":3,"last update":"Mon Jul 10 17:01:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.876,"15min":0.82399999999999995},"min":{"1min":0.66000000000000003,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.2310000000000001,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.84699999999999998,"15min":0.82099999999999995},"min":{"1min":0.69099999999999995,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.159,"5min":2.258,"15min":2.258},"last":1.022}]},{"osd":4,"last update":"Mon Jul 10 17:01:38 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":0.99399999999999999,"15min":0.996},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.0680000000000001,"15min":1.054},"min":{"1min":0.88400000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":1.264}]},{"osd":5,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.0880000000000001,"15min":1.077},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":1.135},{"interface":"front","average":{"1min":1.222,"5min":1.1180000000000001,"15min":1.101},"min":{"1min":0.91700000000000004,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.4810000000000001,"5min":1.782,"15min":1.782},"last":1.234}]},{"osd":6,"last update":"Mon Jul 10 17:01:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.24,"15min":1.236},"min":{"1min":0.82999999999999996,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.552,"5min":1.734,"15min":1.734},"last":1.169},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.226,"15min":1.218},"min":{"1min":0.97299999999999998,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.1890000000000001}]},{"osd":7,"last update":"Mon Jul 10 17:00:42 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":0.999},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":1.105,"5min":1.105,"15min":1.105},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":1.29}]}]},{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5408,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739744,"statfs":{"total":95995035648,"available":95989497856,"internally_reserved":0,"allocated":483328,"data_stored":191358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jul 10 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.71099999999999997,"15min":0.68899999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.1120000000000001,"5min":1.202,"15min":1.202},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.75600000000000001,"15min":0.752},"min":{"1min":0.56799999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.71099999999999997}]},{"osd":1,"last update":"Mon Jul 10 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.79500000000000004,"15min":0.79200000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.77700000000000002,"15min":0.755},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.101,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.66500000000000004}]},{"osd":2,"last update":"Mon Jul 10 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.85599999999999998,"15min":0.83999999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.81699999999999995,"15min":0.81299999999999994},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.74399999999999999}]},{"osd":4,"last update":"Mon Jul 10 17:01:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.96799999999999997,"15min":0.95599999999999996},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.248,"5min":1.369,"15min":1.369},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.069,"5min":1.0429999999999999,"15min":1.0289999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.319,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.0589999999999999}]},{"osd":5,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.048,"15min":1.044},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.073,"5min":1.093,"15min":1.0960000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":1.196}]},{"osd":6,"last update":"Mon Jul 10 17:01:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1259999999999999,"15min":1.1279999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.2270000000000001,"15min":1.2350000000000001},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.544,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":1.032}]},{"osd":7,"last update":"Mon Jul 10 17:00:40 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":1.569,"5min":1.569,"15min":1.569},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":1.2669999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5408,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739744,"statfs":{"total":95995035648,"available":95989497856,"internally_reserved":0,"allocated":483328,"data_stored":191358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jul 10 17:01:37 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.93200000000000005,"15min":0.93600000000000005},"min":{"1min":0.60499999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.92500000000000004,"15min":0.93400000000000005},"min":{"1min":0.66400000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.996}]},{"osd":1,"last update":"Mon Jul 10 17:01:37 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.89800000000000002,"15min":0.88500000000000001},"min":{"1min":0.70399999999999996,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.222,"5min":1.258,"15min":1.258},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.016,"5min":0.97399999999999998,"15min":0.96399999999999997},"min":{"1min":0.70299999999999996,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":1.2450000000000001}]},{"osd":2,"last update":"Mon Jul 10 17:01:37 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.93999999999999995,"15min":0.93400000000000005},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.93300000000000005,"15min":0.90600000000000003},"min":{"1min":0.77800000000000002,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":1.1519999999999999}]},{"osd":3,"last update":"Mon Jul 10 17:01:37 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.99099999999999999,"15min":0.96599999999999997},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.1890000000000001,"5min":2.8519999999999999,"15min":2.8519999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.95599999999999996,"15min":0.93799999999999994},"min":{"1min":0.72199999999999998,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.21,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":1.1100000000000001}]},{"osd":5,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.97199999999999998,"15min":0.96899999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.395,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.048,"5min":1.0189999999999999,"15min":1.014},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.429,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.83799999999999997}]},{"osd":6,"last update":"Mon Jul 10 17:01:19 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.002,"15min":1.0009999999999999},"min":{"1min":0.60399999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.4379999999999999,"5min":1.675,"15min":1.675},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.139,"15min":1.131},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":1.296}]},{"osd":7,"last update":"Mon Jul 10 17:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.2170000000000001,"15min":1.2110000000000001},"min":{"1min":0.94199999999999995,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.341},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.1890000000000001,"15min":1.181},"min":{"1min":0.97099999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.4970000000000001,"5min":1.514,"15min":1.514},"last":1.0820000000000001}]}]},{"osd":5,"up_from":37,"seq":158913789990,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5340,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739812,"statfs":{"total":95995035648,"available":95989567488,"internally_reserved":0,"allocated":479232,"data_stored":191043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.91700000000000004,"15min":0.91400000000000003},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.1850000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":1.012},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.94699999999999995,"15min":0.94399999999999995},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.93200000000000005}]},{"osd":1,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.93200000000000005,"15min":0.93000000000000005},"min":{"1min":0.66000000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.2450000000000001,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.92900000000000005,"15min":0.92000000000000004},"min":{"1min":0.72399999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":1.1259999999999999}]},{"osd":2,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.95799999999999996,"15min":0.95099999999999996},"min":{"1min":0.77700000000000002,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.105},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.93600000000000005,"15min":0.93000000000000005},"min":{"1min":0.70799999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.3400000000000001,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":0.90300000000000002}]},{"osd":3,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.046,"5min":1.014,"15min":1.0089999999999999},"min":{"1min":0.74099999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.74299999999999999}]},{"osd":4,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.90700000000000003,"15min":0.88700000000000001},"min":{"1min":0.752,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":2.3820000000000001,"5min":2.3820000000000001,"15min":2.3820000000000001},"last":1.04},{"interface":"front","average":{"1min":1.05,"5min":0.94299999999999995,"15min":0.92500000000000004},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":2.1829999999999998,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":1.0569999999999999}]},{"osd":6,"last update":"Mon Jul 10 17:01:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.0629999999999999,"15min":1.0649999999999999},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.401,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.179},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.0780000000000001,"15min":1.071},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.7589999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":1.157}]},{"osd":7,"last update":"Mon Jul 10 17:00:42 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":1.0760000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-10T17:01:45.018 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph pg dump --format=json 2023-07-10T17:01:45.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:45 smithi192 bash[23730]: audit 2023-07-10T17:01:44.253381+0000 mgr.y (mgr.14144) 262 : audit [DBG] from='client.14580 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-10T17:01:45.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:45 smithi116 bash[28351]: audit 2023-07-10T17:01:44.253381+0000 mgr.y (mgr.14144) 262 : audit [DBG] from='client.14580 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-10T17:01:45.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:45 smithi116 bash[23740]: audit 2023-07-10T17:01:44.253381+0000 mgr.y (mgr.14144) 262 : audit [DBG] from='client.14580 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-10T17:01:46.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:46 smithi192 bash[23730]: cluster 2023-07-10T17:01:45.453215+0000 mgr.y (mgr.14144) 263 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:46.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:46 smithi116 bash[28351]: cluster 2023-07-10T17:01:45.453215+0000 mgr.y (mgr.14144) 263 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:46.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:46 smithi116 bash[23740]: cluster 2023-07-10T17:01:45.453215+0000 mgr.y (mgr.14144) 263 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:47.765 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:01:48.855 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:48 smithi116 bash[23740]: cluster 2023-07-10T17:01:47.453929+0000 mgr.y (mgr.14144) 264 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:48.856 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:48 smithi116 bash[28351]: cluster 2023-07-10T17:01:47.453929+0000 mgr.y (mgr.14144) 264 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:48.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:48 smithi192 bash[23730]: cluster 2023-07-10T17:01:47.453929+0000 mgr.y (mgr.14144) 264 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:50.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:50 smithi192 bash[23730]: cluster 2023-07-10T17:01:49.454597+0000 mgr.y (mgr.14144) 265 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:50.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:50 smithi116 bash[23740]: cluster 2023-07-10T17:01:49.454597+0000 mgr.y (mgr.14144) 265 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:50.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:50 smithi116 bash[28351]: cluster 2023-07-10T17:01:49.454597+0000 mgr.y (mgr.14144) 265 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:50.903 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-07-10T17:01:50.905 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-07-10T17:01:51.650 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2023-07-10T17:01:49.454293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":43256,"kb_used_data":3768,"kb_used_omap":0,"kb_used_meta":39424,"kb_avail":749917960,"statfs":{"total":767960285184,"available":767915991040,"internally_reserved":0,"allocated":3858432,"data_stored":1530234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40370176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"51","state":"active+clean","last_fresh":"2023-07-10T16:59:41.552040+0000","last_change":"2023-07-10T16:59:41.552040+0000","last_active":"2023-07-10T16:59:41.552040+0000","last_peered":"2023-07-10T16:59:41.552040+0000","last_clean":"2023-07-10T16:59:41.552040+0000","last_became_active":"2023-07-10T16:59:41.551241+0000","last_became_peered":"2023-07-10T16:59:41.551241+0000","last_unstale":"2023-07-10T16:59:41.552040+0000","last_undegraded":"2023-07-10T16:59:41.552040+0000","last_fullsized":"2023-07-10T16:59:41.552040+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-10T16:57:56.919074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-10T16:57:56.919074+0000","last_clean_scrub_stamp":"2023-07-10T16:57:56.919074+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397532,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5216,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93739936,"statfs":{"total":95995035648,"available":95989694464,"internally_reserved":0,"allocated":483328,"data_stored":191358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jul 10 17:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.98699999999999999,"15min":0.98899999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.306,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.97899999999999998,"15min":0.97699999999999998},"min":{"1min":0.74199999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.296,"5min":1.52,"15min":1.52},"last":0.76100000000000001}]},{"osd":1,"last update":"Mon Jul 10 17:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.90700000000000003,"15min":0.89800000000000002},"min":{"1min":0.67600000000000005,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.4159999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.996,"15min":1},"min":{"1min":0.71599999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.329,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.60699999999999998}]},{"osd":2,"last update":"Mon Jul 10 17:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.92300000000000004,"15min":0.91500000000000004},"min":{"1min":0.67300000000000004,"5min":0.628,"15min":0.628},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0229999999999999,"15min":1.026},"min":{"1min":0.747,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.361,"5min":1.581,"15min":1.581},"last":0.92700000000000005}]},{"osd":3,"last update":"Mon Jul 10 17:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.94599999999999995,"15min":0.93999999999999995},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0620000000000001,"15min":1.0569999999999999},"min":{"1min":0.80500000000000005,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.73099999999999998}]},{"osd":4,"last update":"Mon Jul 10 17:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.95999999999999996,"15min":0.96299999999999997},"min":{"1min":0.68999999999999995,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.248,"5min":1.484,"15min":1.484},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.99199999999999999,"15min":0.98799999999999999},"min":{"1min":0.624,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.411,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.80700000000000005}]},{"osd":5,"last update":"Mon Jul 10 17:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.0289999999999999,"15min":1.018},"min":{"1min":0.73099999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.5289999999999999,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.0860000000000001,"15min":1.073},"min":{"1min":0.86499999999999999,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.3480000000000001,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.872}]},{"osd":6,"last update":"Mon Jul 10 17:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.149,"15min":1.145},"min":{"1min":0.94999999999999996,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.389,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.149,"5min":1.109,"15min":1.1020000000000001},"min":{"1min":0.627,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.345,"5min":1.681,"15min":1.681},"last":1.048}]}]},{"osd":6,"up_from":43,"seq":184683593761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5280,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93739872,"statfs":{"total":95995035648,"available":95989628928,"internally_reserved":0,"allocated":483328,"data_stored":191358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jul 10 17:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.3400000000000001,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.94699999999999995,"15min":0.95199999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":0.879}]},{"osd":1,"last update":"Mon Jul 10 17:01:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.016,"15min":1.014},"min":{"1min":0.64900000000000002,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.268,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0880000000000001,"15min":1.0920000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":1.238}]},{"osd":2,"last update":"Mon Jul 10 17:01:17 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.012,"15min":1.0109999999999999},"min":{"1min":0.73699999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.4019999999999999,"5min":1.552,"15min":1.552},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1,"15min":1.0089999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":1.107}]},{"osd":3,"last update":"Mon Jul 10 17:01:17 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0089999999999999,"15min":1.0009999999999999},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.121,"15min":1.131},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.248,"5min":1.635,"15min":1.635},"last":1.071}]},{"osd":4,"last update":"Mon Jul 10 17:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.98499999999999999,"15min":0.99299999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.089,"5min":1.109,"15min":1.1120000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":1.266}]},{"osd":5,"last update":"Mon Jul 10 17:01:17 2023","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.0509999999999999,"15min":1.0609999999999999},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.2709999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1779999999999999,"15min":1.1759999999999999},"min":{"1min":0.86799999999999999,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.5920000000000001,"5min":1.645,"15min":1.645},"last":1.1439999999999999}]},{"osd":7,"last update":"Mon Jul 10 17:01:45 2023","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.1910000000000001,"15min":1.206},"min":{"1min":0.81299999999999994,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.5720000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.175},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.3029999999999999,"15min":1.3069999999999999},"min":{"1min":1.0960000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.5289999999999999,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":1.214}]}]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5532,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739620,"statfs":{"total":95995035648,"available":95989370880,"internally_reserved":0,"allocated":479232,"data_stored":191043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jul 10 17:01:24 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.67500000000000004,"15min":0.60699999999999998},"min":{"1min":0.49299999999999999,"5min":0.442,"15min":0.442},"max":{"1min":1.131,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.72799999999999998,"15min":0.66200000000000003},"min":{"1min":0.443,"5min":0.436,"15min":0.436},"max":{"1min":1.101,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.71199999999999997}]},{"osd":2,"last update":"Mon Jul 10 17:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.77900000000000003,"15min":0.73699999999999999},"min":{"1min":0.55600000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.80700000000000005,"15min":0.76700000000000002},"min":{"1min":0.52800000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.218,"5min":1.228,"15min":1.228},"last":0.80500000000000005}]},{"osd":3,"last update":"Mon Jul 10 17:01:14 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81299999999999994,"15min":0.80500000000000005},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":1.0629999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.84799999999999998,"15min":0.83399999999999996},"min":{"1min":0.64400000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.732,"5min":1.732,"15min":1.732},"last":1.135}]},{"osd":4,"last update":"Mon Jul 10 17:01:32 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.008,"15min":1.004},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.4199999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0409999999999999,"15min":1.0149999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.3879999999999999,"5min":2.1349999999999998,"15min":2.1349999999999998},"last":1.2330000000000001}]},{"osd":5,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.212,"15min":1.23},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.5029999999999999,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.179,"15min":1.1850000000000001},"min":{"1min":0.90000000000000002,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.3819999999999999,"5min":1.431,"15min":1.431},"last":0.94699999999999995}]},{"osd":6,"last update":"Mon Jul 10 17:01:20 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.22,"15min":1.238},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.587,"5min":1.821,"15min":1.821},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.254,"15min":1.268},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.474,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":1.0369999999999999}]},{"osd":7,"last update":"Mon Jul 10 17:01:42 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.294,"15min":1.3100000000000001},"min":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.7030000000000001,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.355},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.276,"15min":1.2989999999999999},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.6859999999999999,"5min":2.198,"15min":2.198},"last":1.169}]}]},{"osd":0,"up_from":8,"seq":34359738431,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5596,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739556,"statfs":{"total":95995035648,"available":95989305344,"internally_reserved":0,"allocated":479232,"data_stored":191043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jul 10 17:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.69599999999999995,"15min":0.64000000000000001},"min":{"1min":0.61399999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1359999999999999,"5min":1.248,"15min":1.248},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.79000000000000004,"15min":0.74399999999999999},"min":{"1min":0.70299999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0249999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.80000000000000004}]},{"osd":2,"last update":"Mon Jul 10 17:00:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.69999999999999996,"15min":0.65900000000000003},"min":{"1min":0.44900000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.0700000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.73999999999999999,"15min":0.70699999999999996},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.628}]},{"osd":3,"last update":"Mon Jul 10 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.748,"15min":0.72699999999999998},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.085,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.91900000000000004,"15min":0.88700000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.3129999999999999,"5min":1.377,"15min":1.377},"last":1.2589999999999999}]},{"osd":4,"last update":"Mon Jul 10 17:01:33 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0669999999999999,"15min":1.077},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.363,"5min":1.466,"15min":1.466},"last":1.337},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0549999999999999,"15min":1.0389999999999999},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.651,"5min":1.651,"15min":1.651},"last":0.86899999999999999}]},{"osd":5,"last update":"Mon Jul 10 17:00:59 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.157,"15min":1.155},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.389},{"interface":"front","average":{"1min":1.109,"5min":1.095,"15min":1.093},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.97599999999999998}]},{"osd":6,"last update":"Mon Jul 10 17:01:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2210000000000001,"15min":1.2230000000000001},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":1.5089999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.214,"15min":1.2170000000000001},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"max":{"1min":1.466,"5min":1.466,"15min":1.466},"last":1.01}]},{"osd":7,"last update":"Mon Jul 10 17:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.25,"15min":1.256},"min":{"1min":0.88700000000000001,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.5669999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.258,"5min":1.214,"15min":1.206},"min":{"1min":0.92000000000000004,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.1899999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5476,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739676,"statfs":{"total":95995035648,"available":95989428224,"internally_reserved":0,"allocated":487424,"data_stored":191673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jul 10 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.66200000000000003,"15min":0.629},"min":{"1min":0.57099999999999995,"5min":0.496,"15min":0.496},"max":{"1min":1.1910000000000001,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84599999999999997,"15min":0.85199999999999998},"min":{"1min":0.61899999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.282,"5min":3.9569999999999999,"15min":3.9569999999999999},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Jul 10 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.83599999999999997,"15min":0.85699999999999998},"min":{"1min":0.58899999999999997,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1000000000000001,"5min":3.9900000000000002,"15min":3.9900000000000002},"last":0.749},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.81999999999999995,"15min":0.77800000000000002},"min":{"1min":0.63800000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.873}]},{"osd":3,"last update":"Mon Jul 10 17:01:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.876,"15min":0.82399999999999995},"min":{"1min":0.66000000000000003,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.2310000000000001,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":1.4710000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.84699999999999998,"15min":0.82099999999999995},"min":{"1min":0.69099999999999995,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.159,"5min":2.258,"15min":2.258},"last":0.91600000000000004}]},{"osd":4,"last update":"Mon Jul 10 17:01:38 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":0.99399999999999999,"15min":0.996},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":1.385},{"interface":"front","average":{"1min":1.141,"5min":1.0680000000000001,"15min":1.054},"min":{"1min":0.88400000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":1.2090000000000001}]},{"osd":5,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.0880000000000001,"15min":1.077},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.222,"5min":1.1180000000000001,"15min":1.101},"min":{"1min":0.91700000000000004,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.4810000000000001,"5min":1.782,"15min":1.782},"last":1.5469999999999999}]},{"osd":6,"last update":"Mon Jul 10 17:01:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.24,"15min":1.236},"min":{"1min":0.82999999999999996,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.552,"5min":1.734,"15min":1.734},"last":1.419},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.226,"15min":1.218},"min":{"1min":0.97299999999999998,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.1699999999999999}]},{"osd":7,"last update":"Mon Jul 10 17:01:43 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.214,"15min":1.218},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.361,"5min":1.337,"15min":1.333},"min":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.3520000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149743,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5408,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739744,"statfs":{"total":95995035648,"available":95989497856,"internally_reserved":0,"allocated":483328,"data_stored":191358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jul 10 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.71099999999999997,"15min":0.68899999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.1120000000000001,"5min":1.202,"15min":1.202},"last":0.751},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.75600000000000001,"15min":0.752},"min":{"1min":0.56799999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.72699999999999998}]},{"osd":1,"last update":"Mon Jul 10 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.79500000000000004,"15min":0.79200000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.77700000000000002,"15min":0.755},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.101,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.61699999999999999}]},{"osd":2,"last update":"Mon Jul 10 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.85599999999999998,"15min":0.83999999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.81699999999999995,"15min":0.81299999999999994},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.70799999999999996}]},{"osd":4,"last update":"Mon Jul 10 17:01:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.96799999999999997,"15min":0.95599999999999996},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.248,"5min":1.369,"15min":1.369},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.069,"5min":1.0429999999999999,"15min":1.0289999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.319,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.85299999999999998}]},{"osd":5,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.048,"15min":1.044},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.073,"5min":1.093,"15min":1.0960000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":1.032}]},{"osd":6,"last update":"Mon Jul 10 17:01:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1259999999999999,"15min":1.1279999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.179,"5min":1.2270000000000001,"15min":1.2350000000000001},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.544,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":1.1950000000000001}]},{"osd":7,"last update":"Mon Jul 10 17:01:42 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.248,"15min":1.2569999999999999},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.518,"5min":1.569,"15min":1.569},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.179,"15min":1.177},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.46,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":1.1200000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5408,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739744,"statfs":{"total":95995035648,"available":95989497856,"internally_reserved":0,"allocated":483328,"data_stored":191358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jul 10 17:01:37 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.93200000000000005,"15min":0.93600000000000005},"min":{"1min":0.60499999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":1.093},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.92500000000000004,"15min":0.93400000000000005},"min":{"1min":0.66400000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.71599999999999997}]},{"osd":1,"last update":"Mon Jul 10 17:01:37 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.89800000000000002,"15min":0.88500000000000001},"min":{"1min":0.70399999999999996,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.222,"5min":1.258,"15min":1.258},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.016,"5min":0.97399999999999998,"15min":0.96399999999999997},"min":{"1min":0.70299999999999996,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":1.2070000000000001}]},{"osd":2,"last update":"Mon Jul 10 17:01:37 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.93999999999999995,"15min":0.93400000000000005},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.1539999999999999},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.93300000000000005,"15min":0.90600000000000003},"min":{"1min":0.77800000000000002,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":1.0149999999999999}]},{"osd":3,"last update":"Mon Jul 10 17:01:37 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.99099999999999999,"15min":0.96599999999999997},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.1890000000000001,"5min":2.8519999999999999,"15min":2.8519999999999999},"last":1.073},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.95599999999999996,"15min":0.93799999999999994},"min":{"1min":0.72199999999999998,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.21,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.83399999999999996}]},{"osd":5,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.97199999999999998,"15min":0.96899999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.395,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.048,"5min":1.0189999999999999,"15min":1.014},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.429,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.90900000000000003}]},{"osd":6,"last update":"Mon Jul 10 17:01:19 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.002,"15min":1.0009999999999999},"min":{"1min":0.60399999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.4379999999999999,"5min":1.675,"15min":1.675},"last":0.873},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.139,"15min":1.131},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":1.294}]},{"osd":7,"last update":"Mon Jul 10 17:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.2170000000000001,"15min":1.2110000000000001},"min":{"1min":0.94199999999999995,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.252},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.1890000000000001,"15min":1.181},"min":{"1min":0.97099999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.4970000000000001,"5min":1.514,"15min":1.514},"last":1.119}]}]},{"osd":5,"up_from":37,"seq":158913789991,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5340,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739812,"statfs":{"total":95995035648,"available":95989567488,"internally_reserved":0,"allocated":479232,"data_stored":191043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.91700000000000004,"15min":0.91400000000000003},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.1850000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":1.3140000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.94699999999999995,"15min":0.94399999999999995},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.76700000000000002}]},{"osd":1,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.93200000000000005,"15min":0.93000000000000005},"min":{"1min":0.66000000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.2450000000000001,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":1.234},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.92900000000000005,"15min":0.92000000000000004},"min":{"1min":0.72399999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":0.90200000000000002}]},{"osd":2,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.95799999999999996,"15min":0.95099999999999996},"min":{"1min":0.77700000000000002,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.876},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.93600000000000005,"15min":0.93000000000000005},"min":{"1min":0.70799999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.3400000000000001,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":1.371}]},{"osd":3,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":1.093},{"interface":"front","average":{"1min":1.046,"5min":1.014,"15min":1.0089999999999999},"min":{"1min":0.74099999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":1.468}]},{"osd":4,"last update":"Mon Jul 10 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.90700000000000003,"15min":0.88700000000000001},"min":{"1min":0.752,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":2.3820000000000001,"5min":2.3820000000000001,"15min":2.3820000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.05,"5min":0.94299999999999995,"15min":0.92500000000000004},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":2.1829999999999998,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":1.218}]},{"osd":6,"last update":"Mon Jul 10 17:01:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.0629999999999999,"15min":1.0649999999999999},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.401,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.0780000000000001,"15min":1.071},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.7589999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":1.153}]},{"osd":7,"last update":"Mon Jul 10 17:01:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.147,"15min":1.1379999999999999},"min":{"1min":0.93600000000000005,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":1.272},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.1599999999999999,"15min":1.1419999999999999},"min":{"1min":0.96799999999999997,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":1.419}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-10T17:01:51.652 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-07-10T17:01:51.652 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-07-10T17:01:51.653 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-07-10T17:01:51.653 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- ceph health --format=json 2023-07-10T17:01:52.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:52 smithi192 bash[23730]: audit 2023-07-10T17:01:50.893588+0000 mgr.y (mgr.14144) 266 : audit [DBG] from='client.14586 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-10T17:01:52.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:52 smithi192 bash[23730]: cluster 2023-07-10T17:01:51.455294+0000 mgr.y (mgr.14144) 267 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:52.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:52 smithi116 bash[23740]: audit 2023-07-10T17:01:50.893588+0000 mgr.y (mgr.14144) 266 : audit [DBG] from='client.14586 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-10T17:01:52.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:52 smithi116 bash[23740]: cluster 2023-07-10T17:01:51.455294+0000 mgr.y (mgr.14144) 267 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:52.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:52 smithi116 bash[28351]: audit 2023-07-10T17:01:50.893588+0000 mgr.y (mgr.14144) 266 : audit [DBG] from='client.14586 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-10T17:01:52.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:52 smithi116 bash[28351]: cluster 2023-07-10T17:01:51.455294+0000 mgr.y (mgr.14144) 267 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:54.394 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/mon.a/config 2023-07-10T17:01:54.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:54 smithi192 bash[23730]: cluster 2023-07-10T17:01:53.455724+0000 mgr.y (mgr.14144) 268 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:54.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:54 smithi116 bash[23740]: cluster 2023-07-10T17:01:53.455724+0000 mgr.y (mgr.14144) 268 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:54.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:54 smithi116 bash[28351]: cluster 2023-07-10T17:01:53.455724+0000 mgr.y (mgr.14144) 268 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:56.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:56 smithi116 bash[23740]: cluster 2023-07-10T17:01:55.456331+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v239: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:56.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:56 smithi116 bash[28351]: cluster 2023-07-10T17:01:55.456331+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v239: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:56.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:56 smithi192 bash[23730]: cluster 2023-07-10T17:01:55.456331+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v239: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:57.679 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-07-10T17:01:58.384 INFO:teuthology.orchestra.run.smithi116.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-07-10T17:01:58.385 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-07-10T17:01:58.386 INFO:tasks.cephadm:Setup complete, yielding 2023-07-10T17:01:58.386 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-10T17:01:58.397 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2023-07-10T17:01:58.399 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2023-07-10T17:01:58.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:58 smithi116 bash[23740]: cluster 2023-07-10T17:01:57.456859+0000 mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:58.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:01:58 smithi116 bash[23740]: audit 2023-07-10T17:01:57.676781+0000 mon.a (mon.0) 520 : audit [DBG] from='client.? 172.21.15.116:0/2346622615' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-10T17:01:58.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:58 smithi116 bash[28351]: cluster 2023-07-10T17:01:57.456859+0000 mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:58.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:01:58 smithi116 bash[28351]: audit 2023-07-10T17:01:57.676781+0000 mon.a (mon.0) 520 : audit [DBG] from='client.? 172.21.15.116:0/2346622615' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-10T17:01:58.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:58 smithi192 bash[23730]: cluster 2023-07-10T17:01:57.456859+0000 mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:01:58.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:01:58 smithi192 bash[23730]: audit 2023-07-10T17:01:57.676781+0000 mon.a (mon.0) 520 : audit [DBG] from='client.? 172.21.15.116:0/2346622615' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-10T17:02:00.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:00 smithi192 bash[23730]: cluster 2023-07-10T17:01:59.457489+0000 mgr.y (mgr.14144) 271 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:00.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:00 smithi116 bash[28351]: cluster 2023-07-10T17:01:59.457489+0000 mgr.y (mgr.14144) 271 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:00.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:00 smithi116 bash[23740]: cluster 2023-07-10T17:01:59.457489+0000 mgr.y (mgr.14144) 271 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:02.406 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-10T17:02:02.419 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2023-07-10T17:02:02.420 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2023-07-10T17:02:02.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:02 smithi116 bash[23740]: cluster 2023-07-10T17:02:01.458051+0000 mgr.y (mgr.14144) 272 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:02.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:02 smithi116 bash[23740]: audit 2023-07-10T17:02:01.679329+0000 mon.a (mon.0) 521 : audit [INF] from='client.? ' entity='client.admin' 2023-07-10T17:02:02.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:02 smithi116 bash[23740]: audit 2023-07-10T17:02:01.696229+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:02.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:02 smithi116 bash[28351]: cluster 2023-07-10T17:02:01.458051+0000 mgr.y (mgr.14144) 272 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:02.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:02 smithi116 bash[28351]: audit 2023-07-10T17:02:01.679329+0000 mon.a (mon.0) 521 : audit [INF] from='client.? ' entity='client.admin' 2023-07-10T17:02:02.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:02 smithi116 bash[28351]: audit 2023-07-10T17:02:01.696229+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:03.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:02 smithi192 bash[23730]: cluster 2023-07-10T17:02:01.458051+0000 mgr.y (mgr.14144) 272 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:03.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:02 smithi192 bash[23730]: audit 2023-07-10T17:02:01.679329+0000 mon.a (mon.0) 521 : audit [INF] from='client.? ' entity='client.admin' 2023-07-10T17:02:03.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:02 smithi192 bash[23730]: audit 2023-07-10T17:02:01.696229+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:04 smithi192 bash[23730]: cluster 2023-07-10T17:02:03.458508+0000 mgr.y (mgr.14144) 273 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:04.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:04 smithi116 bash[23740]: cluster 2023-07-10T17:02:03.458508+0000 mgr.y (mgr.14144) 273 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:04.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:04 smithi116 bash[28351]: cluster 2023-07-10T17:02:03.458508+0000 mgr.y (mgr.14144) 273 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:06.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:06 smithi192 bash[23730]: cluster 2023-07-10T17:02:05.459029+0000 mgr.y (mgr.14144) 274 : cluster [DBG] pgmap v244: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:06.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:06 smithi116 bash[28351]: cluster 2023-07-10T17:02:05.459029+0000 mgr.y (mgr.14144) 274 : cluster [DBG] pgmap v244: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:06.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:06 smithi116 bash[23740]: cluster 2023-07-10T17:02:05.459029+0000 mgr.y (mgr.14144) 274 : cluster [DBG] pgmap v244: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:07.533 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:02:07.533 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9", 2023-07-10T17:02:07.533 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "r", 2023-07-10T17:02:07.533 INFO:teuthology.orchestra.run.smithi116.stdout: "current_period": "b1eee481-0d82-4738-91c9-d10ef9ba7bdc", 2023-07-10T17:02:07.534 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1 2023-07-10T17:02:07.534 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:02:07.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:07 smithi192 bash[23730]: cluster 2023-07-10T17:02:06.486653+0000 mon.a (mon.0) 523 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2023-07-10T17:02:07.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:07 smithi192 bash[23730]: audit 2023-07-10T17:02:06.492259+0000 mon.c (mon.1) 25 : audit [INF] from='client.? 172.21.15.116:0/2257102364' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-10T17:02:07.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:07 smithi192 bash[23730]: audit 2023-07-10T17:02:06.493837+0000 mon.a (mon.0) 524 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-10T17:02:07.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:07 smithi116 bash[23740]: cluster 2023-07-10T17:02:06.486653+0000 mon.a (mon.0) 523 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2023-07-10T17:02:07.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:07 smithi116 bash[23740]: audit 2023-07-10T17:02:06.492259+0000 mon.c (mon.1) 25 : audit [INF] from='client.? 172.21.15.116:0/2257102364' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-10T17:02:07.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:07 smithi116 bash[23740]: audit 2023-07-10T17:02:06.493837+0000 mon.a (mon.0) 524 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-10T17:02:07.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:07 smithi116 bash[28351]: cluster 2023-07-10T17:02:06.486653+0000 mon.a (mon.0) 523 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2023-07-10T17:02:07.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:07 smithi116 bash[28351]: audit 2023-07-10T17:02:06.492259+0000 mon.c (mon.1) 25 : audit [INF] from='client.? 172.21.15.116:0/2257102364' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-10T17:02:07.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:07 smithi116 bash[28351]: audit 2023-07-10T17:02:06.493837+0000 mon.a (mon.0) 524 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-10T17:02:08.301 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2023-07-10T17:02:08.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:08 smithi116 bash[28351]: cluster 2023-07-10T17:02:07.459704+0000 mgr.y (mgr.14144) 275 : cluster [DBG] pgmap v246: 33 pgs: 4 creating+peering, 1 active+clean, 28 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:08.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:08 smithi116 bash[28351]: audit 2023-07-10T17:02:07.491733+0000 mon.a (mon.0) 525 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2023-07-10T17:02:08.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:08 smithi116 bash[28351]: cluster 2023-07-10T17:02:07.492075+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2023-07-10T17:02:08.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:08 smithi116 bash[23740]: cluster 2023-07-10T17:02:07.459704+0000 mgr.y (mgr.14144) 275 : cluster [DBG] pgmap v246: 33 pgs: 4 creating+peering, 1 active+clean, 28 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:08.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:08 smithi116 bash[23740]: audit 2023-07-10T17:02:07.491733+0000 mon.a (mon.0) 525 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2023-07-10T17:02:08.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:08 smithi116 bash[23740]: cluster 2023-07-10T17:02:07.492075+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2023-07-10T17:02:08.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:08 smithi192 bash[23730]: cluster 2023-07-10T17:02:07.459704+0000 mgr.y (mgr.14144) 275 : cluster [DBG] pgmap v246: 33 pgs: 4 creating+peering, 1 active+clean, 28 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:08.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:08 smithi192 bash[23730]: audit 2023-07-10T17:02:07.491733+0000 mon.a (mon.0) 525 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2023-07-10T17:02:08.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:08 smithi192 bash[23730]: cluster 2023-07-10T17:02:07.492075+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2023-07-10T17:02:09.813 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:09 smithi116 bash[23740]: cluster 2023-07-10T17:02:08.507621+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2023-07-10T17:02:09.813 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:09 smithi116 bash[28351]: cluster 2023-07-10T17:02:08.507621+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2023-07-10T17:02:09.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:09 smithi192 bash[23730]: cluster 2023-07-10T17:02:08.507621+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2023-07-10T17:02:10.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:10 smithi192 bash[23730]: cluster 2023-07-10T17:02:09.460342+0000 mgr.y (mgr.14144) 276 : cluster [DBG] pgmap v249: 33 pgs: 5 creating+activating, 4 creating+peering, 4 active+clean, 20 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:10.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:10 smithi116 bash[23740]: cluster 2023-07-10T17:02:09.460342+0000 mgr.y (mgr.14144) 276 : cluster [DBG] pgmap v249: 33 pgs: 5 creating+activating, 4 creating+peering, 4 active+clean, 20 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:10.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:10 smithi116 bash[28351]: cluster 2023-07-10T17:02:09.460342+0000 mgr.y (mgr.14144) 276 : cluster [DBG] pgmap v249: 33 pgs: 5 creating+activating, 4 creating+peering, 4 active+clean, 20 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:11.407 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:02:11.407 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "f8259aee-7892-4e80-84d3-d38407c5bc73", 2023-07-10T17:02:11.407 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default", 2023-07-10T17:02:11.408 INFO:teuthology.orchestra.run.smithi116.stdout: "api_name": "default", 2023-07-10T17:02:11.408 INFO:teuthology.orchestra.run.smithi116.stdout: "is_master": "true", 2023-07-10T17:02:11.408 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [], 2023-07-10T17:02:11.408 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames": [], 2023-07-10T17:02:11.408 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames_s3website": [], 2023-07-10T17:02:11.408 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "", 2023-07-10T17:02:11.408 INFO:teuthology.orchestra.run.smithi116.stdout: "zones": [], 2023-07-10T17:02:11.409 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_targets": [], 2023-07-10T17:02:11.409 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "", 2023-07-10T17:02:11.409 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9", 2023-07-10T17:02:11.409 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_policy": { 2023-07-10T17:02:11.409 INFO:teuthology.orchestra.run.smithi116.stdout: "groups": [] 2023-07-10T17:02:11.409 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:02:11.409 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:02:12.128 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2023-07-10T17:02:12.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:12 smithi116 bash[28351]: cluster 2023-07-10T17:02:11.461062+0000 mgr.y (mgr.14144) 277 : cluster [DBG] pgmap v250: 33 pgs: 5 creating+activating, 4 creating+peering, 24 active+clean; 470 B data, 43 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:12.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:12 smithi116 bash[23740]: cluster 2023-07-10T17:02:11.461062+0000 mgr.y (mgr.14144) 277 : cluster [DBG] pgmap v250: 33 pgs: 5 creating+activating, 4 creating+peering, 24 active+clean; 470 B data, 43 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:12.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:12 smithi192 bash[23730]: cluster 2023-07-10T17:02:11.461062+0000 mgr.y (mgr.14144) 277 : cluster [DBG] pgmap v250: 33 pgs: 5 creating+activating, 4 creating+peering, 24 active+clean; 470 B data, 43 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:02:14.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:14 smithi192 bash[23730]: cluster 2023-07-10T17:02:13.461723+0000 mgr.y (mgr.14144) 278 : cluster [DBG] pgmap v251: 33 pgs: 33 active+clean; 562 B data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.5 KiB/s wr, 2 op/s 2023-07-10T17:02:14.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:14 smithi116 bash[23740]: cluster 2023-07-10T17:02:13.461723+0000 mgr.y (mgr.14144) 278 : cluster [DBG] pgmap v251: 33 pgs: 33 active+clean; 562 B data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.5 KiB/s wr, 2 op/s 2023-07-10T17:02:14.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:14 smithi116 bash[28351]: cluster 2023-07-10T17:02:13.461723+0000 mgr.y (mgr.14144) 278 : cluster [DBG] pgmap v251: 33 pgs: 33 active+clean; 562 B data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.5 KiB/s wr, 2 op/s 2023-07-10T17:02:15.341 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:02:15.342 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "22dab653-5cb2-4177-bc15-3abd20c5b1ce", 2023-07-10T17:02:15.342 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "z", 2023-07-10T17:02:15.342 INFO:teuthology.orchestra.run.smithi116.stdout: "domain_root": "z.rgw.meta:root", 2023-07-10T17:02:15.342 INFO:teuthology.orchestra.run.smithi116.stdout: "control_pool": "z.rgw.control", 2023-07-10T17:02:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "gc_pool": "z.rgw.log:gc", 2023-07-10T17:02:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "lc_pool": "z.rgw.log:lc", 2023-07-10T17:02:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "log_pool": "z.rgw.log", 2023-07-10T17:02:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "intent_log_pool": "z.rgw.log:intent", 2023-07-10T17:02:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "usage_log_pool": "z.rgw.log:usage", 2023-07-10T17:02:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "roles_pool": "z.rgw.meta:roles", 2023-07-10T17:02:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "reshard_pool": "z.rgw.log:reshard", 2023-07-10T17:02:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2023-07-10T17:02:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "user_email_pool": "z.rgw.meta:users.email", 2023-07-10T17:02:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2023-07-10T17:02:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2023-07-10T17:02:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "otp_pool": "z.rgw.otp", 2023-07-10T17:02:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "system_key": { 2023-07-10T17:02:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "access_key": "", 2023-07-10T17:02:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "secret_key": "" 2023-07-10T17:02:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:02:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_pools": [ 2023-07-10T17:02:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-07-10T17:02:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "default-placement", 2023-07-10T17:02:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2023-07-10T17:02:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "index_pool": "z.rgw.buckets.index", 2023-07-10T17:02:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": { 2023-07-10T17:02:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD": { 2023-07-10T17:02:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "data_pool": "z.rgw.buckets.data" 2023-07-10T17:02:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:02:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:02:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2023-07-10T17:02:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "index_type": 0 2023-07-10T17:02:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:02:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:02:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:02:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9", 2023-07-10T17:02:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "notif_pool": "z.rgw.log:notif" 2023-07-10T17:02:15.348 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:02:16.030 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2023-07-10T17:02:16.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:16 smithi192 bash[23730]: cluster 2023-07-10T17:02:15.462485+0000 mgr.y (mgr.14144) 279 : cluster [DBG] pgmap v252: 33 pgs: 33 active+clean; 772 B data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.2 KiB/s wr, 1 op/s 2023-07-10T17:02:16.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:16 smithi116 bash[23740]: cluster 2023-07-10T17:02:15.462485+0000 mgr.y (mgr.14144) 279 : cluster [DBG] pgmap v252: 33 pgs: 33 active+clean; 772 B data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.2 KiB/s wr, 1 op/s 2023-07-10T17:02:16.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:16 smithi116 bash[28351]: cluster 2023-07-10T17:02:15.462485+0000 mgr.y (mgr.14144) 279 : cluster [DBG] pgmap v252: 33 pgs: 33 active+clean; 772 B data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.2 KiB/s wr, 1 op/s 2023-07-10T17:02:18.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:18 smithi192 bash[23730]: cluster 2023-07-10T17:02:17.463296+0000 mgr.y (mgr.14144) 280 : cluster [DBG] pgmap v253: 33 pgs: 33 active+clean; 772 B data, 43 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 853 B/s wr, 1 op/s 2023-07-10T17:02:18.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:18 smithi116 bash[23740]: cluster 2023-07-10T17:02:17.463296+0000 mgr.y (mgr.14144) 280 : cluster [DBG] pgmap v253: 33 pgs: 33 active+clean; 772 B data, 43 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 853 B/s wr, 1 op/s 2023-07-10T17:02:18.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:18 smithi116 bash[28351]: cluster 2023-07-10T17:02:17.463296+0000 mgr.y (mgr.14144) 280 : cluster [DBG] pgmap v253: 33 pgs: 33 active+clean; 772 B data, 43 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 853 B/s wr, 1 op/s 2023-07-10T17:02:20.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:19 smithi192 bash[23730]: audit 2023-07-10T17:02:19.609439+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2023-07-10T17:02:20.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:19 smithi192 bash[23730]: audit 2023-07-10T17:02:19.610945+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:02:20.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:19 smithi192 bash[23730]: audit 2023-07-10T17:02:19.611560+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:02:20.142 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:19 smithi116 bash[23740]: audit 2023-07-10T17:02:19.609439+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2023-07-10T17:02:20.142 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:19 smithi116 bash[23740]: audit 2023-07-10T17:02:19.610945+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:02:20.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:19 smithi116 bash[23740]: audit 2023-07-10T17:02:19.611560+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:02:20.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:19 smithi116 bash[28351]: audit 2023-07-10T17:02:19.609439+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2023-07-10T17:02:20.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:19 smithi116 bash[28351]: audit 2023-07-10T17:02:19.610945+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:02:20.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:19 smithi116 bash[28351]: audit 2023-07-10T17:02:19.611560+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:02:21.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:20 smithi192 bash[23730]: cluster 2023-07-10T17:02:19.463953+0000 mgr.y (mgr.14144) 281 : cluster [DBG] pgmap v254: 33 pgs: 33 active+clean; 1.5 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 767 B/s wr, 1 op/s 2023-07-10T17:02:21.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:20 smithi192 bash[23730]: audit 2023-07-10T17:02:19.669461+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2023-07-10T17:02:21.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:20 smithi192 bash[23730]: cluster 2023-07-10T17:02:19.669812+0000 mon.a (mon.0) 532 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2023-07-10T17:02:21.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:20 smithi192 bash[23730]: audit 2023-07-10T17:02:19.673837+0000 mon.a (mon.0) 533 : audit [INF] from='client.? 172.21.15.116:0/3002584689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2023-07-10T17:02:21.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:20 smithi192 bash[23730]: audit 2023-07-10T17:02:20.673477+0000 mon.a (mon.0) 534 : audit [INF] from='client.? 172.21.15.116:0/3002584689' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2023-07-10T17:02:21.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:20 smithi192 bash[23730]: cluster 2023-07-10T17:02:20.674576+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2023-07-10T17:02:21.142 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:20 smithi116 bash[23740]: cluster 2023-07-10T17:02:19.463953+0000 mgr.y (mgr.14144) 281 : cluster [DBG] pgmap v254: 33 pgs: 33 active+clean; 1.5 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 767 B/s wr, 1 op/s 2023-07-10T17:02:21.142 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:20 smithi116 bash[23740]: audit 2023-07-10T17:02:19.669461+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2023-07-10T17:02:21.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:20 smithi116 bash[23740]: cluster 2023-07-10T17:02:19.669812+0000 mon.a (mon.0) 532 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2023-07-10T17:02:21.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:20 smithi116 bash[23740]: audit 2023-07-10T17:02:19.673837+0000 mon.a (mon.0) 533 : audit [INF] from='client.? 172.21.15.116:0/3002584689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2023-07-10T17:02:21.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:20 smithi116 bash[23740]: audit 2023-07-10T17:02:20.673477+0000 mon.a (mon.0) 534 : audit [INF] from='client.? 172.21.15.116:0/3002584689' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2023-07-10T17:02:21.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:20 smithi116 bash[23740]: cluster 2023-07-10T17:02:20.674576+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2023-07-10T17:02:21.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:20 smithi116 bash[28351]: cluster 2023-07-10T17:02:19.463953+0000 mgr.y (mgr.14144) 281 : cluster [DBG] pgmap v254: 33 pgs: 33 active+clean; 1.5 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 767 B/s wr, 1 op/s 2023-07-10T17:02:21.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:20 smithi116 bash[28351]: audit 2023-07-10T17:02:19.669461+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2023-07-10T17:02:21.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:20 smithi116 bash[28351]: cluster 2023-07-10T17:02:19.669812+0000 mon.a (mon.0) 532 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2023-07-10T17:02:21.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:20 smithi116 bash[28351]: audit 2023-07-10T17:02:19.673837+0000 mon.a (mon.0) 533 : audit [INF] from='client.? 172.21.15.116:0/3002584689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2023-07-10T17:02:21.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:20 smithi116 bash[28351]: audit 2023-07-10T17:02:20.673477+0000 mon.a (mon.0) 534 : audit [INF] from='client.? 172.21.15.116:0/3002584689' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2023-07-10T17:02:21.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:20 smithi116 bash[28351]: cluster 2023-07-10T17:02:20.674576+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2023-07-10T17:02:23.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:22 smithi192 bash[23730]: cluster 2023-07-10T17:02:21.464698+0000 mgr.y (mgr.14144) 282 : cluster [DBG] pgmap v257: 65 pgs: 24 unknown, 41 active+clean; 1.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 895 B/s wr, 3 op/s 2023-07-10T17:02:23.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:22 smithi192 bash[23730]: cluster 2023-07-10T17:02:21.677631+0000 mon.a (mon.0) 536 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2023-07-10T17:02:23.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:22 smithi192 bash[23730]: audit 2023-07-10T17:02:21.680119+0000 mon.a (mon.0) 537 : audit [INF] from='client.? 172.21.15.116:0/3002584689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2023-07-10T17:02:23.142 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:22 smithi116 bash[23740]: cluster 2023-07-10T17:02:21.464698+0000 mgr.y (mgr.14144) 282 : cluster [DBG] pgmap v257: 65 pgs: 24 unknown, 41 active+clean; 1.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 895 B/s wr, 3 op/s 2023-07-10T17:02:23.142 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:22 smithi116 bash[23740]: cluster 2023-07-10T17:02:21.677631+0000 mon.a (mon.0) 536 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2023-07-10T17:02:23.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:22 smithi116 bash[23740]: audit 2023-07-10T17:02:21.680119+0000 mon.a (mon.0) 537 : audit [INF] from='client.? 172.21.15.116:0/3002584689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2023-07-10T17:02:23.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:22 smithi116 bash[28351]: cluster 2023-07-10T17:02:21.464698+0000 mgr.y (mgr.14144) 282 : cluster [DBG] pgmap v257: 65 pgs: 24 unknown, 41 active+clean; 1.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 895 B/s wr, 3 op/s 2023-07-10T17:02:23.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:22 smithi116 bash[28351]: cluster 2023-07-10T17:02:21.677631+0000 mon.a (mon.0) 536 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2023-07-10T17:02:23.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:22 smithi116 bash[28351]: audit 2023-07-10T17:02:21.680119+0000 mon.a (mon.0) 537 : audit [INF] from='client.? 172.21.15.116:0/3002584689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2023-07-10T17:02:24.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:24 smithi192 bash[23730]: audit 2023-07-10T17:02:22.680845+0000 mon.a (mon.0) 538 : audit [INF] from='client.? 172.21.15.116:0/3002584689' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2023-07-10T17:02:24.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:24 smithi192 bash[23730]: cluster 2023-07-10T17:02:22.681779+0000 mon.a (mon.0) 539 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2023-07-10T17:02:24.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:24 smithi116 bash[23740]: audit 2023-07-10T17:02:22.680845+0000 mon.a (mon.0) 538 : audit [INF] from='client.? 172.21.15.116:0/3002584689' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2023-07-10T17:02:24.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:24 smithi116 bash[23740]: cluster 2023-07-10T17:02:22.681779+0000 mon.a (mon.0) 539 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2023-07-10T17:02:24.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:24 smithi116 bash[28351]: audit 2023-07-10T17:02:22.680845+0000 mon.a (mon.0) 538 : audit [INF] from='client.? 172.21.15.116:0/3002584689' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2023-07-10T17:02:24.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:24 smithi116 bash[28351]: cluster 2023-07-10T17:02:22.681779+0000 mon.a (mon.0) 539 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2023-07-10T17:02:25.385 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:25 smithi192 bash[23730]: cluster 2023-07-10T17:02:23.465368+0000 mgr.y (mgr.14144) 283 : cluster [DBG] pgmap v260: 97 pgs: 3 creating+peering, 24 unknown, 70 active+clean; 5.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 31 op/s 2023-07-10T17:02:25.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:25 smithi192 bash[23730]: cluster 2023-07-10T17:02:24.021224+0000 mon.a (mon.0) 540 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2023-07-10T17:02:25.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:25 smithi192 bash[23730]: audit 2023-07-10T17:02:24.024571+0000 mon.c (mon.1) 26 : audit [INF] from='client.? 172.21.15.116:0/215251782' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2023-07-10T17:02:25.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:25 smithi192 bash[23730]: audit 2023-07-10T17:02:24.030197+0000 mon.a (mon.0) 541 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2023-07-10T17:02:25.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:25 smithi192 bash[23730]: audit 2023-07-10T17:02:25.022368+0000 mon.a (mon.0) 542 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2023-07-10T17:02:25.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:25 smithi192 bash[23730]: cluster 2023-07-10T17:02:25.022697+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2023-07-10T17:02:25.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:25 smithi116 bash[23740]: cluster 2023-07-10T17:02:23.465368+0000 mgr.y (mgr.14144) 283 : cluster [DBG] pgmap v260: 97 pgs: 3 creating+peering, 24 unknown, 70 active+clean; 5.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 31 op/s 2023-07-10T17:02:25.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:25 smithi116 bash[23740]: cluster 2023-07-10T17:02:24.021224+0000 mon.a (mon.0) 540 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2023-07-10T17:02:25.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:25 smithi116 bash[23740]: audit 2023-07-10T17:02:24.024571+0000 mon.c (mon.1) 26 : audit [INF] from='client.? 172.21.15.116:0/215251782' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2023-07-10T17:02:25.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:25 smithi116 bash[23740]: audit 2023-07-10T17:02:24.030197+0000 mon.a (mon.0) 541 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2023-07-10T17:02:25.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:25 smithi116 bash[23740]: audit 2023-07-10T17:02:25.022368+0000 mon.a (mon.0) 542 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2023-07-10T17:02:25.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:25 smithi116 bash[23740]: cluster 2023-07-10T17:02:25.022697+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2023-07-10T17:02:25.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:25 smithi116 bash[28351]: cluster 2023-07-10T17:02:23.465368+0000 mgr.y (mgr.14144) 283 : cluster [DBG] pgmap v260: 97 pgs: 3 creating+peering, 24 unknown, 70 active+clean; 5.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 31 op/s 2023-07-10T17:02:25.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:25 smithi116 bash[28351]: cluster 2023-07-10T17:02:24.021224+0000 mon.a (mon.0) 540 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2023-07-10T17:02:25.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:25 smithi116 bash[28351]: audit 2023-07-10T17:02:24.024571+0000 mon.c (mon.1) 26 : audit [INF] from='client.? 172.21.15.116:0/215251782' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2023-07-10T17:02:25.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:25 smithi116 bash[28351]: audit 2023-07-10T17:02:24.030197+0000 mon.a (mon.0) 541 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2023-07-10T17:02:25.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:25 smithi116 bash[28351]: audit 2023-07-10T17:02:25.022368+0000 mon.a (mon.0) 542 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2023-07-10T17:02:25.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:25 smithi116 bash[28351]: cluster 2023-07-10T17:02:25.022697+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2023-07-10T17:02:26.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:26 smithi192 bash[23730]: audit 2023-07-10T17:02:25.032807+0000 mon.c (mon.1) 27 : audit [INF] from='client.? 172.21.15.116:0/215251782' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-10T17:02:26.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:26 smithi192 bash[23730]: audit 2023-07-10T17:02:25.042670+0000 mon.a (mon.0) 544 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-10T17:02:26.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:26 smithi192 bash[23730]: cluster 2023-07-10T17:02:25.466161+0000 mgr.y (mgr.14144) 284 : cluster [DBG] pgmap v263: 129 pgs: 11 creating+peering, 34 unknown, 84 active+clean; 11 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 15 KiB/s wr, 59 op/s; 11 B/s, 0 objects/s recovering 2023-07-10T17:02:26.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:26 smithi192 bash[23730]: audit 2023-07-10T17:02:26.024865+0000 mon.a (mon.0) 545 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-07-10T17:02:26.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:26 smithi192 bash[23730]: cluster 2023-07-10T17:02:26.025068+0000 mon.a (mon.0) 546 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2023-07-10T17:02:26.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:26 smithi192 bash[23730]: audit 2023-07-10T17:02:26.026350+0000 mon.c (mon.1) 28 : audit [INF] from='client.? 172.21.15.116:0/215251782' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2023-07-10T17:02:26.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:26 smithi192 bash[23730]: audit 2023-07-10T17:02:26.026910+0000 mon.a (mon.0) 547 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2023-07-10T17:02:26.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:26 smithi116 bash[23740]: audit 2023-07-10T17:02:25.032807+0000 mon.c (mon.1) 27 : audit [INF] from='client.? 172.21.15.116:0/215251782' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-10T17:02:26.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:26 smithi116 bash[23740]: audit 2023-07-10T17:02:25.042670+0000 mon.a (mon.0) 544 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-10T17:02:26.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:26 smithi116 bash[23740]: cluster 2023-07-10T17:02:25.466161+0000 mgr.y (mgr.14144) 284 : cluster [DBG] pgmap v263: 129 pgs: 11 creating+peering, 34 unknown, 84 active+clean; 11 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 15 KiB/s wr, 59 op/s; 11 B/s, 0 objects/s recovering 2023-07-10T17:02:26.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:26 smithi116 bash[23740]: audit 2023-07-10T17:02:26.024865+0000 mon.a (mon.0) 545 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-07-10T17:02:26.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:26 smithi116 bash[23740]: cluster 2023-07-10T17:02:26.025068+0000 mon.a (mon.0) 546 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2023-07-10T17:02:26.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:26 smithi116 bash[23740]: audit 2023-07-10T17:02:26.026350+0000 mon.c (mon.1) 28 : audit [INF] from='client.? 172.21.15.116:0/215251782' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2023-07-10T17:02:26.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:26 smithi116 bash[23740]: audit 2023-07-10T17:02:26.026910+0000 mon.a (mon.0) 547 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2023-07-10T17:02:26.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:26 smithi116 bash[28351]: audit 2023-07-10T17:02:25.032807+0000 mon.c (mon.1) 27 : audit [INF] from='client.? 172.21.15.116:0/215251782' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-10T17:02:26.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:26 smithi116 bash[28351]: audit 2023-07-10T17:02:25.042670+0000 mon.a (mon.0) 544 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-10T17:02:26.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:26 smithi116 bash[28351]: cluster 2023-07-10T17:02:25.466161+0000 mgr.y (mgr.14144) 284 : cluster [DBG] pgmap v263: 129 pgs: 11 creating+peering, 34 unknown, 84 active+clean; 11 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 15 KiB/s wr, 59 op/s; 11 B/s, 0 objects/s recovering 2023-07-10T17:02:26.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:26 smithi116 bash[28351]: audit 2023-07-10T17:02:26.024865+0000 mon.a (mon.0) 545 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-07-10T17:02:26.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:26 smithi116 bash[28351]: cluster 2023-07-10T17:02:26.025068+0000 mon.a (mon.0) 546 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2023-07-10T17:02:26.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:26 smithi116 bash[28351]: audit 2023-07-10T17:02:26.026350+0000 mon.c (mon.1) 28 : audit [INF] from='client.? 172.21.15.116:0/215251782' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2023-07-10T17:02:26.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:26 smithi116 bash[28351]: audit 2023-07-10T17:02:26.026910+0000 mon.a (mon.0) 547 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2023-07-10T17:02:27.224 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:02:27.225 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "181137c9-eca1-4790-9c0f-8c483c2547bd", 2023-07-10T17:02:27.225 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2023-07-10T17:02:27.226 INFO:teuthology.orchestra.run.smithi116.stdout: "predecessor_uuid": "b1eee481-0d82-4738-91c9-d10ef9ba7bdc", 2023-07-10T17:02:27.226 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": [], 2023-07-10T17:02:27.226 INFO:teuthology.orchestra.run.smithi116.stdout: "period_map": { 2023-07-10T17:02:27.226 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "181137c9-eca1-4790-9c0f-8c483c2547bd", 2023-07-10T17:02:27.227 INFO:teuthology.orchestra.run.smithi116.stdout: "zonegroups": [ 2023-07-10T17:02:27.227 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-07-10T17:02:27.227 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "f8259aee-7892-4e80-84d3-d38407c5bc73", 2023-07-10T17:02:27.227 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default", 2023-07-10T17:02:27.227 INFO:teuthology.orchestra.run.smithi116.stdout: "api_name": "default", 2023-07-10T17:02:27.227 INFO:teuthology.orchestra.run.smithi116.stdout: "is_master": "true", 2023-07-10T17:02:27.227 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [], 2023-07-10T17:02:27.228 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames": [], 2023-07-10T17:02:27.228 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames_s3website": [], 2023-07-10T17:02:27.228 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "22dab653-5cb2-4177-bc15-3abd20c5b1ce", 2023-07-10T17:02:27.228 INFO:teuthology.orchestra.run.smithi116.stdout: "zones": [ 2023-07-10T17:02:27.228 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-07-10T17:02:27.229 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "22dab653-5cb2-4177-bc15-3abd20c5b1ce", 2023-07-10T17:02:27.229 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "z", 2023-07-10T17:02:27.229 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [], 2023-07-10T17:02:27.229 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2023-07-10T17:02:27.229 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "false", 2023-07-10T17:02:27.230 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2023-07-10T17:02:27.230 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2023-07-10T17:02:27.230 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2023-07-10T17:02:27.230 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "true", 2023-07-10T17:02:27.230 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2023-07-10T17:02:27.231 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2023-07-10T17:02:27.231 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:02:27.231 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:02:27.231 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_targets": [ 2023-07-10T17:02:27.231 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-07-10T17:02:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default-placement", 2023-07-10T17:02:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2023-07-10T17:02:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": [ 2023-07-10T17:02:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD" 2023-07-10T17:02:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2023-07-10T17:02:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:02:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:02:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "default-placement", 2023-07-10T17:02:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9", 2023-07-10T17:02:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_policy": { 2023-07-10T17:02:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: "groups": [] 2023-07-10T17:02:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:02:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:02:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:02:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: "short_zone_ids": [ 2023-07-10T17:02:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-07-10T17:02:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "22dab653-5cb2-4177-bc15-3abd20c5b1ce", 2023-07-10T17:02:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 3066537749 2023-07-10T17:02:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:02:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2023-07-10T17:02:27.236 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:02:27.236 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zonegroup": "f8259aee-7892-4e80-84d3-d38407c5bc73", 2023-07-10T17:02:27.236 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "22dab653-5cb2-4177-bc15-3abd20c5b1ce", 2023-07-10T17:02:27.236 INFO:teuthology.orchestra.run.smithi116.stdout: "period_config": { 2023-07-10T17:02:27.237 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_quota": { 2023-07-10T17:02:27.237 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2023-07-10T17:02:27.237 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2023-07-10T17:02:27.237 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2023-07-10T17:02:27.237 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2023-07-10T17:02:27.238 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2023-07-10T17:02:27.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:02:27.238 INFO:teuthology.orchestra.run.smithi116.stdout: "user_quota": { 2023-07-10T17:02:27.238 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2023-07-10T17:02:27.238 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2023-07-10T17:02:27.238 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2023-07-10T17:02:27.238 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2023-07-10T17:02:27.239 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2023-07-10T17:02:27.239 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:02:27.239 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:02:27.239 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9", 2023-07-10T17:02:27.239 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_name": "r", 2023-07-10T17:02:27.240 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2023-07-10T17:02:27.240 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:02:28.121 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2023-07-10T17:02:28.303 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:28 smithi116 bash[23740]: audit 2023-07-10T17:02:27.026901+0000 mon.a (mon.0) 548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2023-07-10T17:02:28.303 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:28 smithi116 bash[23740]: cluster 2023-07-10T17:02:27.027227+0000 mon.a (mon.0) 549 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2023-07-10T17:02:28.303 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:28 smithi116 bash[23740]: cluster 2023-07-10T17:02:27.467030+0000 mgr.y (mgr.14144) 285 : cluster [DBG] pgmap v266: 129 pgs: 10 creating+peering, 12 unknown, 107 active+clean; 17 KiB data, 46 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 18 KiB/s wr, 72 op/s; 11 B/s, 0 objects/s recovering 2023-07-10T17:02:28.303 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:28 smithi116 bash[28351]: audit 2023-07-10T17:02:27.026901+0000 mon.a (mon.0) 548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2023-07-10T17:02:28.304 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:28 smithi116 bash[28351]: cluster 2023-07-10T17:02:27.027227+0000 mon.a (mon.0) 549 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2023-07-10T17:02:28.304 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:28 smithi116 bash[28351]: cluster 2023-07-10T17:02:27.467030+0000 mgr.y (mgr.14144) 285 : cluster [DBG] pgmap v266: 129 pgs: 10 creating+peering, 12 unknown, 107 active+clean; 17 KiB data, 46 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 18 KiB/s wr, 72 op/s; 11 B/s, 0 objects/s recovering 2023-07-10T17:02:28.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:28 smithi192 bash[23730]: audit 2023-07-10T17:02:27.026901+0000 mon.a (mon.0) 548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2023-07-10T17:02:28.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:28 smithi192 bash[23730]: cluster 2023-07-10T17:02:27.027227+0000 mon.a (mon.0) 549 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2023-07-10T17:02:28.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:28 smithi192 bash[23730]: cluster 2023-07-10T17:02:27.467030+0000 mgr.y (mgr.14144) 285 : cluster [DBG] pgmap v266: 129 pgs: 10 creating+peering, 12 unknown, 107 active+clean; 17 KiB data, 46 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 18 KiB/s wr, 72 op/s; 11 B/s, 0 objects/s recovering 2023-07-10T17:02:30.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:30 smithi192 bash[23730]: cluster 2023-07-10T17:02:29.468145+0000 mgr.y (mgr.14144) 286 : cluster [DBG] pgmap v267: 129 pgs: 8 creating+peering, 121 active+clean; 24 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 21 KiB/s wr, 84 op/s; 8 B/s, 0 objects/s recovering 2023-07-10T17:02:30.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:30 smithi116 bash[23740]: cluster 2023-07-10T17:02:29.468145+0000 mgr.y (mgr.14144) 286 : cluster [DBG] pgmap v267: 129 pgs: 8 creating+peering, 121 active+clean; 24 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 21 KiB/s wr, 84 op/s; 8 B/s, 0 objects/s recovering 2023-07-10T17:02:30.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:30 smithi116 bash[28351]: cluster 2023-07-10T17:02:29.468145+0000 mgr.y (mgr.14144) 286 : cluster [DBG] pgmap v267: 129 pgs: 8 creating+peering, 121 active+clean; 24 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 21 KiB/s wr, 84 op/s; 8 B/s, 0 objects/s recovering 2023-07-10T17:02:31.483 INFO:teuthology.orchestra.run.smithi116.stdout:Scheduled rgw.foo update... 2023-07-10T17:02:32.223 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph orch apply rgw smpl' 2023-07-10T17:02:32.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:32 smithi192 bash[23730]: cluster 2023-07-10T17:02:31.469315+0000 mgr.y (mgr.14144) 287 : cluster [DBG] pgmap v268: 129 pgs: 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 17 KiB/s wr, 64 op/s 2023-07-10T17:02:32.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:32 smithi192 bash[23730]: audit 2023-07-10T17:02:31.473963+0000 mgr.y (mgr.14144) 288 : audit [DBG] from='client.14661 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:02:32.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:32 smithi192 bash[23730]: cephadm 2023-07-10T17:02:31.475342+0000 mgr.y (mgr.14144) 289 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-07-10T17:02:32.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:32 smithi192 bash[23730]: audit 2023-07-10T17:02:31.479726+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:32 smithi192 bash[23730]: audit 2023-07-10T17:02:31.751462+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:32 smithi192 bash[23730]: audit 2023-07-10T17:02:31.763282+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:32 smithi192 bash[23730]: audit 2023-07-10T17:02:31.770268+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:32 smithi192 bash[23730]: audit 2023-07-10T17:02:31.776975+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:32 smithi192 bash[23730]: audit 2023-07-10T17:02:31.777972+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.myrhsb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:02:32.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:32 smithi192 bash[23730]: audit 2023-07-10T17:02:31.782415+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.myrhsb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-10T17:02:32.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:32 smithi192 bash[23730]: audit 2023-07-10T17:02:31.790981+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:32 smithi192 bash[23730]: audit 2023-07-10T17:02:31.792320+0000 mon.a (mon.0) 558 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:02:32.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[23740]: cluster 2023-07-10T17:02:31.469315+0000 mgr.y (mgr.14144) 287 : cluster [DBG] pgmap v268: 129 pgs: 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 17 KiB/s wr, 64 op/s 2023-07-10T17:02:32.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[23740]: audit 2023-07-10T17:02:31.473963+0000 mgr.y (mgr.14144) 288 : audit [DBG] from='client.14661 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:02:32.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[23740]: cephadm 2023-07-10T17:02:31.475342+0000 mgr.y (mgr.14144) 289 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-07-10T17:02:32.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[23740]: audit 2023-07-10T17:02:31.479726+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[23740]: audit 2023-07-10T17:02:31.751462+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[23740]: audit 2023-07-10T17:02:31.763282+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[23740]: audit 2023-07-10T17:02:31.770268+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[23740]: audit 2023-07-10T17:02:31.776975+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[23740]: audit 2023-07-10T17:02:31.777972+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.myrhsb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:02:32.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[23740]: audit 2023-07-10T17:02:31.782415+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.myrhsb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-10T17:02:32.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[23740]: audit 2023-07-10T17:02:31.790981+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[23740]: audit 2023-07-10T17:02:31.792320+0000 mon.a (mon.0) 558 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:02:32.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[28351]: cluster 2023-07-10T17:02:31.469315+0000 mgr.y (mgr.14144) 287 : cluster [DBG] pgmap v268: 129 pgs: 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 17 KiB/s wr, 64 op/s 2023-07-10T17:02:32.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[28351]: audit 2023-07-10T17:02:31.473963+0000 mgr.y (mgr.14144) 288 : audit [DBG] from='client.14661 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:02:32.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[28351]: cephadm 2023-07-10T17:02:31.475342+0000 mgr.y (mgr.14144) 289 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-07-10T17:02:32.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[28351]: audit 2023-07-10T17:02:31.479726+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[28351]: audit 2023-07-10T17:02:31.751462+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[28351]: audit 2023-07-10T17:02:31.763282+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[28351]: audit 2023-07-10T17:02:31.770268+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[28351]: audit 2023-07-10T17:02:31.776975+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[28351]: audit 2023-07-10T17:02:31.777972+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.myrhsb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:02:32.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[28351]: audit 2023-07-10T17:02:31.782415+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.myrhsb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-10T17:02:32.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[28351]: audit 2023-07-10T17:02:31.790981+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:32.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:32 smithi116 bash[28351]: audit 2023-07-10T17:02:31.792320+0000 mon.a (mon.0) 558 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:02:33.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:33 smithi116 bash[23740]: cephadm 2023-07-10T17:02:31.771768+0000 mgr.y (mgr.14144) 290 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-07-10T17:02:33.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:33 smithi116 bash[23740]: cephadm 2023-07-10T17:02:31.793605+0000 mgr.y (mgr.14144) 291 : cephadm [INF] Deploying daemon rgw.foo.smithi192.myrhsb on smithi192 2023-07-10T17:02:33.736 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:33 smithi116 bash[28351]: cephadm 2023-07-10T17:02:31.771768+0000 mgr.y (mgr.14144) 290 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-07-10T17:02:33.736 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:33 smithi116 bash[28351]: cephadm 2023-07-10T17:02:31.793605+0000 mgr.y (mgr.14144) 291 : cephadm [INF] Deploying daemon rgw.foo.smithi192.myrhsb on smithi192 2023-07-10T17:02:33.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:33 smithi192 bash[23730]: cephadm 2023-07-10T17:02:31.771768+0000 mgr.y (mgr.14144) 290 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-07-10T17:02:33.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:33 smithi192 bash[23730]: cephadm 2023-07-10T17:02:31.793605+0000 mgr.y (mgr.14144) 291 : cephadm [INF] Deploying daemon rgw.foo.smithi192.myrhsb on smithi192 2023-07-10T17:02:34.576 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:34 smithi192 bash[23730]: cluster 2023-07-10T17:02:33.470100+0000 mgr.y (mgr.14144) 292 : cluster [DBG] pgmap v269: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 9.4 KiB/s rd, 13 KiB/s wr, 52 op/s 2023-07-10T17:02:34.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:34 smithi116 bash[23740]: cluster 2023-07-10T17:02:33.470100+0000 mgr.y (mgr.14144) 292 : cluster [DBG] pgmap v269: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 9.4 KiB/s rd, 13 KiB/s wr, 52 op/s 2023-07-10T17:02:34.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:34 smithi116 bash[28351]: cluster 2023-07-10T17:02:33.470100+0000 mgr.y (mgr.14144) 292 : cluster [DBG] pgmap v269: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 9.4 KiB/s rd, 13 KiB/s wr, 52 op/s 2023-07-10T17:02:35.837 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:35 smithi116 bash[28351]: audit 2023-07-10T17:02:34.553426+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:35.838 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:35 smithi116 bash[28351]: audit 2023-07-10T17:02:34.555445+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi116.stnxgn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:02:35.838 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:35 smithi116 bash[28351]: audit 2023-07-10T17:02:34.560802+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi116.stnxgn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-10T17:02:35.838 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:35 smithi116 bash[28351]: audit 2023-07-10T17:02:34.566438+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:35.838 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:35 smithi116 bash[28351]: audit 2023-07-10T17:02:34.567758+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:02:35.838 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:35 smithi116 bash[28351]: cephadm 2023-07-10T17:02:34.568916+0000 mgr.y (mgr.14144) 293 : cephadm [INF] Deploying daemon rgw.foo.smithi116.stnxgn on smithi116 2023-07-10T17:02:35.838 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:35 smithi116 bash[23740]: audit 2023-07-10T17:02:34.553426+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:35.839 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:35 smithi116 bash[23740]: audit 2023-07-10T17:02:34.555445+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi116.stnxgn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:02:35.839 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:35 smithi116 bash[23740]: audit 2023-07-10T17:02:34.560802+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi116.stnxgn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-10T17:02:35.839 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:35 smithi116 bash[23740]: audit 2023-07-10T17:02:34.566438+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:35.839 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:35 smithi116 bash[23740]: audit 2023-07-10T17:02:34.567758+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:02:35.839 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:35 smithi116 bash[23740]: cephadm 2023-07-10T17:02:34.568916+0000 mgr.y (mgr.14144) 293 : cephadm [INF] Deploying daemon rgw.foo.smithi116.stnxgn on smithi116 2023-07-10T17:02:35.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:35 smithi192 bash[23730]: audit 2023-07-10T17:02:34.553426+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:35.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:35 smithi192 bash[23730]: audit 2023-07-10T17:02:34.555445+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi116.stnxgn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:02:35.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:35 smithi192 bash[23730]: audit 2023-07-10T17:02:34.560802+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi116.stnxgn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-10T17:02:35.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:35 smithi192 bash[23730]: audit 2023-07-10T17:02:34.566438+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:35.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:35 smithi192 bash[23730]: audit 2023-07-10T17:02:34.567758+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:02:35.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:35 smithi192 bash[23730]: cephadm 2023-07-10T17:02:34.568916+0000 mgr.y (mgr.14144) 293 : cephadm [INF] Deploying daemon rgw.foo.smithi116.stnxgn on smithi116 2023-07-10T17:02:36.230 INFO:teuthology.orchestra.run.smithi116.stdout:Scheduled rgw.smpl update... 2023-07-10T17:02:36.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:36 smithi116 bash[23740]: cluster 2023-07-10T17:02:35.471073+0000 mgr.y (mgr.14144) 294 : cluster [DBG] pgmap v270: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 11 KiB/s wr, 44 op/s 2023-07-10T17:02:36.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:36 smithi116 bash[23740]: audit 2023-07-10T17:02:36.227226+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:36.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:36 smithi116 bash[28351]: cluster 2023-07-10T17:02:35.471073+0000 mgr.y (mgr.14144) 294 : cluster [DBG] pgmap v270: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 11 KiB/s wr, 44 op/s 2023-07-10T17:02:36.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:36 smithi116 bash[28351]: audit 2023-07-10T17:02:36.227226+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:36.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:36 smithi192 bash[23730]: cluster 2023-07-10T17:02:35.471073+0000 mgr.y (mgr.14144) 294 : cluster [DBG] pgmap v270: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 11 KiB/s wr, 44 op/s 2023-07-10T17:02:36.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:36 smithi192 bash[23730]: audit 2023-07-10T17:02:36.227226+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:37.455 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph osd pool create foo' 2023-07-10T17:02:37.872 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:37 smithi116 bash[28351]: audit 2023-07-10T17:02:36.222138+0000 mgr.y (mgr.14144) 295 : audit [DBG] from='client.24604 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:02:37.873 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:37 smithi116 bash[28351]: cephadm 2023-07-10T17:02:36.223026+0000 mgr.y (mgr.14144) 296 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-07-10T17:02:37.873 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:37 smithi116 bash[23740]: audit 2023-07-10T17:02:36.222138+0000 mgr.y (mgr.14144) 295 : audit [DBG] from='client.24604 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:02:37.873 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:37 smithi116 bash[23740]: cephadm 2023-07-10T17:02:36.223026+0000 mgr.y (mgr.14144) 296 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-07-10T17:02:37.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:37 smithi192 bash[23730]: audit 2023-07-10T17:02:36.222138+0000 mgr.y (mgr.14144) 295 : audit [DBG] from='client.24604 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:02:37.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:37 smithi192 bash[23730]: cephadm 2023-07-10T17:02:36.223026+0000 mgr.y (mgr.14144) 296 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-07-10T17:02:38.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:38 smithi192 bash[23730]: cluster 2023-07-10T17:02:37.471948+0000 mgr.y (mgr.14144) 297 : cluster [DBG] pgmap v271: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 6.4 KiB/s wr, 51 op/s 2023-07-10T17:02:38.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:38 smithi192 bash[23730]: audit 2023-07-10T17:02:37.905516+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:38.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:38 smithi116 bash[23740]: cluster 2023-07-10T17:02:37.471948+0000 mgr.y (mgr.14144) 297 : cluster [DBG] pgmap v271: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 6.4 KiB/s wr, 51 op/s 2023-07-10T17:02:38.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:38 smithi116 bash[23740]: audit 2023-07-10T17:02:37.905516+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:38.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:38 smithi116 bash[28351]: cluster 2023-07-10T17:02:37.471948+0000 mgr.y (mgr.14144) 297 : cluster [DBG] pgmap v271: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 6.4 KiB/s wr, 51 op/s 2023-07-10T17:02:38.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:38 smithi116 bash[28351]: audit 2023-07-10T17:02:37.905516+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:40.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:40 smithi192 bash[23730]: cluster 2023-07-10T17:02:39.472998+0000 mgr.y (mgr.14144) 298 : cluster [DBG] pgmap v272: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 5.6 KiB/s wr, 115 op/s 2023-07-10T17:02:40.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:40 smithi116 bash[23740]: cluster 2023-07-10T17:02:39.472998+0000 mgr.y (mgr.14144) 298 : cluster [DBG] pgmap v272: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 5.6 KiB/s wr, 115 op/s 2023-07-10T17:02:40.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:40 smithi116 bash[28351]: cluster 2023-07-10T17:02:39.472998+0000 mgr.y (mgr.14144) 298 : cluster [DBG] pgmap v272: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 5.6 KiB/s wr, 115 op/s 2023-07-10T17:02:42.625 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'foo' created 2023-07-10T17:02:42.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:42 smithi192 bash[23730]: cluster 2023-07-10T17:02:41.473878+0000 mgr.y (mgr.14144) 299 : cluster [DBG] pgmap v273: 129 pgs: 129 active+clean; 27 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 2.1 KiB/s wr, 221 op/s 2023-07-10T17:02:42.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:42 smithi192 bash[23730]: audit 2023-07-10T17:02:41.801674+0000 mon.a (mon.0) 566 : audit [INF] from='client.? 172.21.15.116:0/3326130343' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2023-07-10T17:02:42.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:42 smithi192 bash[23730]: audit 2023-07-10T17:02:42.303385+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:42.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:42 smithi116 bash[28351]: cluster 2023-07-10T17:02:41.473878+0000 mgr.y (mgr.14144) 299 : cluster [DBG] pgmap v273: 129 pgs: 129 active+clean; 27 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 2.1 KiB/s wr, 221 op/s 2023-07-10T17:02:42.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:42 smithi116 bash[28351]: audit 2023-07-10T17:02:41.801674+0000 mon.a (mon.0) 566 : audit [INF] from='client.? 172.21.15.116:0/3326130343' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2023-07-10T17:02:42.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:42 smithi116 bash[28351]: audit 2023-07-10T17:02:42.303385+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:42.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:42 smithi116 bash[23740]: cluster 2023-07-10T17:02:41.473878+0000 mgr.y (mgr.14144) 299 : cluster [DBG] pgmap v273: 129 pgs: 129 active+clean; 27 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 2.1 KiB/s wr, 221 op/s 2023-07-10T17:02:42.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:42 smithi116 bash[23740]: audit 2023-07-10T17:02:41.801674+0000 mon.a (mon.0) 566 : audit [INF] from='client.? 172.21.15.116:0/3326130343' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2023-07-10T17:02:42.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:42 smithi116 bash[23740]: audit 2023-07-10T17:02:42.303385+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:43.679 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'rbd pool init foo' 2023-07-10T17:02:43.863 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[28351]: audit 2023-07-10T17:02:42.591737+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:43.864 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[28351]: audit 2023-07-10T17:02:42.611605+0000 mon.a (mon.0) 569 : audit [INF] from='client.? 172.21.15.116:0/3326130343' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2023-07-10T17:02:43.864 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[28351]: cluster 2023-07-10T17:02:42.611697+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2023-07-10T17:02:43.864 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[28351]: audit 2023-07-10T17:02:42.846408+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:43.865 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[28351]: audit 2023-07-10T17:02:42.857717+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:43.865 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[28351]: audit 2023-07-10T17:02:42.858854+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi116.vpoihf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:02:43.865 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[28351]: audit 2023-07-10T17:02:42.866320+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi116.vpoihf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-10T17:02:43.865 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[28351]: audit 2023-07-10T17:02:42.876659+0000 mon.a (mon.0) 575 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:43.865 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[28351]: audit 2023-07-10T17:02:42.878363+0000 mon.a (mon.0) 576 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:02:43.866 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[23740]: audit 2023-07-10T17:02:42.591737+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:43.866 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[23740]: audit 2023-07-10T17:02:42.611605+0000 mon.a (mon.0) 569 : audit [INF] from='client.? 172.21.15.116:0/3326130343' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2023-07-10T17:02:43.866 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[23740]: cluster 2023-07-10T17:02:42.611697+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2023-07-10T17:02:43.866 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[23740]: audit 2023-07-10T17:02:42.846408+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:43.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[23740]: audit 2023-07-10T17:02:42.857717+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:43.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[23740]: audit 2023-07-10T17:02:42.858854+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi116.vpoihf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:02:43.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[23740]: audit 2023-07-10T17:02:42.866320+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi116.vpoihf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-10T17:02:43.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[23740]: audit 2023-07-10T17:02:42.876659+0000 mon.a (mon.0) 575 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:43.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:43 smithi116 bash[23740]: audit 2023-07-10T17:02:42.878363+0000 mon.a (mon.0) 576 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:02:43.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:43 smithi192 bash[23730]: audit 2023-07-10T17:02:42.591737+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:43.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:43 smithi192 bash[23730]: audit 2023-07-10T17:02:42.611605+0000 mon.a (mon.0) 569 : audit [INF] from='client.? 172.21.15.116:0/3326130343' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2023-07-10T17:02:43.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:43 smithi192 bash[23730]: cluster 2023-07-10T17:02:42.611697+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2023-07-10T17:02:43.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:43 smithi192 bash[23730]: audit 2023-07-10T17:02:42.846408+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:43.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:43 smithi192 bash[23730]: audit 2023-07-10T17:02:42.857717+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:43.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:43 smithi192 bash[23730]: audit 2023-07-10T17:02:42.858854+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi116.vpoihf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:02:43.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:43 smithi192 bash[23730]: audit 2023-07-10T17:02:42.866320+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi116.vpoihf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-10T17:02:43.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:43 smithi192 bash[23730]: audit 2023-07-10T17:02:42.876659+0000 mon.a (mon.0) 575 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:43.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:43 smithi192 bash[23730]: audit 2023-07-10T17:02:42.878363+0000 mon.a (mon.0) 576 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:02:44.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:44 smithi192 bash[23730]: cephadm 2023-07-10T17:02:42.851265+0000 mgr.y (mgr.14144) 300 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-07-10T17:02:44.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:44 smithi192 bash[23730]: cephadm 2023-07-10T17:02:42.879817+0000 mgr.y (mgr.14144) 301 : cephadm [INF] Deploying daemon rgw.smpl.smithi116.vpoihf on smithi116 2023-07-10T17:02:44.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:44 smithi192 bash[23730]: cluster 2023-07-10T17:02:43.474664+0000 mgr.y (mgr.14144) 302 : cluster [DBG] pgmap v275: 161 pgs: 10 creating+peering, 22 unknown, 129 active+clean; 27 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 177 KiB/s rd, 102 B/s wr, 342 op/s 2023-07-10T17:02:44.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:44 smithi192 bash[23730]: cluster 2023-07-10T17:02:43.613063+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2023-07-10T17:02:44.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:44 smithi116 bash[23740]: cephadm 2023-07-10T17:02:42.851265+0000 mgr.y (mgr.14144) 300 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-07-10T17:02:44.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:44 smithi116 bash[23740]: cephadm 2023-07-10T17:02:42.879817+0000 mgr.y (mgr.14144) 301 : cephadm [INF] Deploying daemon rgw.smpl.smithi116.vpoihf on smithi116 2023-07-10T17:02:44.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:44 smithi116 bash[23740]: cluster 2023-07-10T17:02:43.474664+0000 mgr.y (mgr.14144) 302 : cluster [DBG] pgmap v275: 161 pgs: 10 creating+peering, 22 unknown, 129 active+clean; 27 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 177 KiB/s rd, 102 B/s wr, 342 op/s 2023-07-10T17:02:44.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:44 smithi116 bash[23740]: cluster 2023-07-10T17:02:43.613063+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2023-07-10T17:02:44.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:44 smithi116 bash[28351]: cephadm 2023-07-10T17:02:42.851265+0000 mgr.y (mgr.14144) 300 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-07-10T17:02:44.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:44 smithi116 bash[28351]: cephadm 2023-07-10T17:02:42.879817+0000 mgr.y (mgr.14144) 301 : cephadm [INF] Deploying daemon rgw.smpl.smithi116.vpoihf on smithi116 2023-07-10T17:02:44.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:44 smithi116 bash[28351]: cluster 2023-07-10T17:02:43.474664+0000 mgr.y (mgr.14144) 302 : cluster [DBG] pgmap v275: 161 pgs: 10 creating+peering, 22 unknown, 129 active+clean; 27 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 177 KiB/s rd, 102 B/s wr, 342 op/s 2023-07-10T17:02:44.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:44 smithi116 bash[28351]: cluster 2023-07-10T17:02:43.613063+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2023-07-10T17:02:45.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:45 smithi192 bash[23730]: cluster 2023-07-10T17:02:44.627622+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2023-07-10T17:02:45.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:45 smithi116 bash[23740]: cluster 2023-07-10T17:02:44.627622+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2023-07-10T17:02:45.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:45 smithi116 bash[28351]: cluster 2023-07-10T17:02:44.627622+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2023-07-10T17:02:46.798 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:46 smithi116 bash[28351]: cluster 2023-07-10T17:02:45.475439+0000 mgr.y (mgr.14144) 303 : cluster [DBG] pgmap v278: 161 pgs: 10 creating+peering, 13 unknown, 138 active+clean; 27 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 265 KiB/s rd, 341 B/s wr, 485 op/s 2023-07-10T17:02:46.799 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:46 smithi116 bash[23740]: cluster 2023-07-10T17:02:45.475439+0000 mgr.y (mgr.14144) 303 : cluster [DBG] pgmap v278: 161 pgs: 10 creating+peering, 13 unknown, 138 active+clean; 27 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 265 KiB/s rd, 341 B/s wr, 485 op/s 2023-07-10T17:02:47.093 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:46 smithi192 bash[23730]: cluster 2023-07-10T17:02:45.475439+0000 mgr.y (mgr.14144) 303 : cluster [DBG] pgmap v278: 161 pgs: 10 creating+peering, 13 unknown, 138 active+clean; 27 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 265 KiB/s rd, 341 B/s wr, 485 op/s 2023-07-10T17:02:48.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:47 smithi192 bash[23730]: audit 2023-07-10T17:02:46.830226+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:48.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:47 smithi192 bash[23730]: audit 2023-07-10T17:02:46.831428+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.yqkmva", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:02:48.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:47 smithi192 bash[23730]: audit 2023-07-10T17:02:46.835784+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.yqkmva", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-10T17:02:48.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:47 smithi192 bash[23730]: audit 2023-07-10T17:02:46.842001+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:48.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:47 smithi192 bash[23730]: audit 2023-07-10T17:02:46.843281+0000 mon.a (mon.0) 583 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:02:48.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:47 smithi192 bash[23730]: cephadm 2023-07-10T17:02:46.844175+0000 mgr.y (mgr.14144) 304 : cephadm [INF] Deploying daemon rgw.smpl.smithi192.yqkmva on smithi192 2023-07-10T17:02:48.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:47 smithi192 bash[23730]: cluster 2023-07-10T17:02:47.476561+0000 mgr.y (mgr.14144) 305 : cluster [DBG] pgmap v279: 161 pgs: 10 creating+peering, 4 unknown, 147 active+clean; 27 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 170 B/s wr, 246 op/s 2023-07-10T17:02:48.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:47 smithi116 bash[23740]: audit 2023-07-10T17:02:46.830226+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:48.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:47 smithi116 bash[23740]: audit 2023-07-10T17:02:46.831428+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.yqkmva", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:02:48.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:47 smithi116 bash[23740]: audit 2023-07-10T17:02:46.835784+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.yqkmva", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-10T17:02:48.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:47 smithi116 bash[23740]: audit 2023-07-10T17:02:46.842001+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:48.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:47 smithi116 bash[23740]: audit 2023-07-10T17:02:46.843281+0000 mon.a (mon.0) 583 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:02:48.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:47 smithi116 bash[23740]: cephadm 2023-07-10T17:02:46.844175+0000 mgr.y (mgr.14144) 304 : cephadm [INF] Deploying daemon rgw.smpl.smithi192.yqkmva on smithi192 2023-07-10T17:02:48.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:47 smithi116 bash[23740]: cluster 2023-07-10T17:02:47.476561+0000 mgr.y (mgr.14144) 305 : cluster [DBG] pgmap v279: 161 pgs: 10 creating+peering, 4 unknown, 147 active+clean; 27 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 170 B/s wr, 246 op/s 2023-07-10T17:02:48.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:47 smithi116 bash[28351]: audit 2023-07-10T17:02:46.830226+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:48.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:47 smithi116 bash[28351]: audit 2023-07-10T17:02:46.831428+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.yqkmva", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:02:48.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:47 smithi116 bash[28351]: audit 2023-07-10T17:02:46.835784+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.yqkmva", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-10T17:02:48.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:47 smithi116 bash[28351]: audit 2023-07-10T17:02:46.842001+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:48.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:47 smithi116 bash[28351]: audit 2023-07-10T17:02:46.843281+0000 mon.a (mon.0) 583 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:02:48.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:47 smithi116 bash[28351]: cephadm 2023-07-10T17:02:46.844175+0000 mgr.y (mgr.14144) 304 : cephadm [INF] Deploying daemon rgw.smpl.smithi192.yqkmva on smithi192 2023-07-10T17:02:48.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:47 smithi116 bash[28351]: cluster 2023-07-10T17:02:47.476561+0000 mgr.y (mgr.14144) 305 : cluster [DBG] pgmap v279: 161 pgs: 10 creating+peering, 4 unknown, 147 active+clean; 27 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 170 B/s wr, 246 op/s 2023-07-10T17:02:49.132 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:48 smithi192 bash[23730]: audit 2023-07-10T17:02:48.294187+0000 mon.a (mon.0) 584 : audit [INF] from='client.? 172.21.15.116:0/1649371071' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2023-07-10T17:02:49.142 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:48 smithi116 bash[23740]: audit 2023-07-10T17:02:48.294187+0000 mon.a (mon.0) 584 : audit [INF] from='client.? 172.21.15.116:0/1649371071' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2023-07-10T17:02:49.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:48 smithi116 bash[28351]: audit 2023-07-10T17:02:48.294187+0000 mon.a (mon.0) 584 : audit [INF] from='client.? 172.21.15.116:0/1649371071' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2023-07-10T17:02:50.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:49 smithi192 bash[23730]: audit 2023-07-10T17:02:48.860297+0000 mon.a (mon.0) 585 : audit [INF] from='client.? 172.21.15.116:0/1649371071' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2023-07-10T17:02:50.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:49 smithi192 bash[23730]: cluster 2023-07-10T17:02:48.860442+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2023-07-10T17:02:50.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:49 smithi192 bash[23730]: cluster 2023-07-10T17:02:49.477382+0000 mgr.y (mgr.14144) 306 : cluster [DBG] pgmap v281: 161 pgs: 161 active+clean; 27 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 170 B/s wr, 103 op/s 2023-07-10T17:02:50.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:49 smithi192 bash[23730]: audit 2023-07-10T17:02:49.688119+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:50.142 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:49 smithi116 bash[23740]: audit 2023-07-10T17:02:48.860297+0000 mon.a (mon.0) 585 : audit [INF] from='client.? 172.21.15.116:0/1649371071' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2023-07-10T17:02:50.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:49 smithi116 bash[23740]: cluster 2023-07-10T17:02:48.860442+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2023-07-10T17:02:50.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:49 smithi116 bash[23740]: cluster 2023-07-10T17:02:49.477382+0000 mgr.y (mgr.14144) 306 : cluster [DBG] pgmap v281: 161 pgs: 161 active+clean; 27 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 170 B/s wr, 103 op/s 2023-07-10T17:02:50.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:49 smithi116 bash[23740]: audit 2023-07-10T17:02:49.688119+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:50.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:49 smithi116 bash[28351]: audit 2023-07-10T17:02:48.860297+0000 mon.a (mon.0) 585 : audit [INF] from='client.? 172.21.15.116:0/1649371071' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2023-07-10T17:02:50.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:49 smithi116 bash[28351]: cluster 2023-07-10T17:02:48.860442+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2023-07-10T17:02:50.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:49 smithi116 bash[28351]: cluster 2023-07-10T17:02:49.477382+0000 mgr.y (mgr.14144) 306 : cluster [DBG] pgmap v281: 161 pgs: 161 active+clean; 27 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 170 B/s wr, 103 op/s 2023-07-10T17:02:50.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:49 smithi116 bash[28351]: audit 2023-07-10T17:02:49.688119+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:51.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:50 smithi192 bash[23730]: cluster 2023-07-10T17:02:49.862256+0000 mon.a (mon.0) 588 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2023-07-10T17:02:51.142 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:50 smithi116 bash[23740]: cluster 2023-07-10T17:02:49.862256+0000 mon.a (mon.0) 588 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2023-07-10T17:02:51.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:50 smithi116 bash[28351]: cluster 2023-07-10T17:02:49.862256+0000 mon.a (mon.0) 588 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2023-07-10T17:02:51.739 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph orch apply iscsi foo u p' 2023-07-10T17:02:52.142 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:51 smithi116 bash[23740]: cluster 2023-07-10T17:02:50.882355+0000 mon.a (mon.0) 589 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2023-07-10T17:02:52.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:51 smithi116 bash[23740]: cluster 2023-07-10T17:02:51.478523+0000 mgr.y (mgr.14144) 307 : cluster [DBG] pgmap v284: 161 pgs: 161 active+clean; 27 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 170 B/s wr, 163 op/s 2023-07-10T17:02:52.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:51 smithi116 bash[28351]: cluster 2023-07-10T17:02:50.882355+0000 mon.a (mon.0) 589 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2023-07-10T17:02:52.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:51 smithi116 bash[28351]: cluster 2023-07-10T17:02:51.478523+0000 mgr.y (mgr.14144) 307 : cluster [DBG] pgmap v284: 161 pgs: 161 active+clean; 27 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 170 B/s wr, 163 op/s 2023-07-10T17:02:52.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:51 smithi192 bash[23730]: cluster 2023-07-10T17:02:50.882355+0000 mon.a (mon.0) 589 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2023-07-10T17:02:52.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:51 smithi192 bash[23730]: cluster 2023-07-10T17:02:51.478523+0000 mgr.y (mgr.14144) 307 : cluster [DBG] pgmap v284: 161 pgs: 161 active+clean; 27 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 170 B/s wr, 163 op/s 2023-07-10T17:02:54.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:54 smithi192 bash[23730]: cluster 2023-07-10T17:02:53.479392+0000 mgr.y (mgr.14144) 308 : cluster [DBG] pgmap v285: 161 pgs: 161 active+clean; 27 KiB data, 86 MiB used, 715 GiB / 715 GiB avail; 250 KiB/s rd, 511 B/s wr, 444 op/s 2023-07-10T17:02:54.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:54 smithi192 bash[23730]: audit 2023-07-10T17:02:54.261933+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:54.888 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:54 smithi116 bash[28351]: cluster 2023-07-10T17:02:53.479392+0000 mgr.y (mgr.14144) 308 : cluster [DBG] pgmap v285: 161 pgs: 161 active+clean; 27 KiB data, 86 MiB used, 715 GiB / 715 GiB avail; 250 KiB/s rd, 511 B/s wr, 444 op/s 2023-07-10T17:02:54.888 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:54 smithi116 bash[28351]: audit 2023-07-10T17:02:54.261933+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:54.888 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:54 smithi116 bash[23740]: cluster 2023-07-10T17:02:53.479392+0000 mgr.y (mgr.14144) 308 : cluster [DBG] pgmap v285: 161 pgs: 161 active+clean; 27 KiB data, 86 MiB used, 715 GiB / 715 GiB avail; 250 KiB/s rd, 511 B/s wr, 444 op/s 2023-07-10T17:02:54.889 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:54 smithi116 bash[23740]: audit 2023-07-10T17:02:54.261933+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:55.310 INFO:teuthology.orchestra.run.smithi116.stdout:Scheduled iscsi.foo update... 2023-07-10T17:02:56.064 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:55 smithi116 bash[28351]: audit 2023-07-10T17:02:54.676989+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:56.065 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:55 smithi116 bash[28351]: audit 2023-07-10T17:02:54.688346+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:56.065 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:55 smithi116 bash[28351]: audit 2023-07-10T17:02:54.706224+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:56.065 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:55 smithi116 bash[28351]: audit 2023-07-10T17:02:55.307093+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:56.066 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:55 smithi116 bash[28351]: audit 2023-07-10T17:02:55.314941+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:56.066 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:55 smithi116 bash[28351]: audit 2023-07-10T17:02:55.329572+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:02:56.066 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:55 smithi116 bash[28351]: audit 2023-07-10T17:02:55.336343+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2023-07-10T17:02:56.067 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:55 smithi116 bash[28351]: audit 2023-07-10T17:02:55.344459+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:02:56.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:55 smithi116 bash[23740]: audit 2023-07-10T17:02:54.676989+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:56.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:55 smithi116 bash[23740]: audit 2023-07-10T17:02:54.688346+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:56.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:55 smithi116 bash[23740]: audit 2023-07-10T17:02:54.706224+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:56.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:55 smithi116 bash[23740]: audit 2023-07-10T17:02:55.307093+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:56.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:55 smithi116 bash[23740]: audit 2023-07-10T17:02:55.314941+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:56.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:55 smithi116 bash[23740]: audit 2023-07-10T17:02:55.329572+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:02:56.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:55 smithi116 bash[23740]: audit 2023-07-10T17:02:55.336343+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2023-07-10T17:02:56.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:55 smithi116 bash[23740]: audit 2023-07-10T17:02:55.344459+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:02:56.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:55 smithi192 bash[23730]: audit 2023-07-10T17:02:54.676989+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:56.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:55 smithi192 bash[23730]: audit 2023-07-10T17:02:54.688346+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:56.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:55 smithi192 bash[23730]: audit 2023-07-10T17:02:54.706224+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:56.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:55 smithi192 bash[23730]: audit 2023-07-10T17:02:55.307093+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:56.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:55 smithi192 bash[23730]: audit 2023-07-10T17:02:55.314941+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:02:56.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:55 smithi192 bash[23730]: audit 2023-07-10T17:02:55.329572+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:02:56.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:55 smithi192 bash[23730]: audit 2023-07-10T17:02:55.336343+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2023-07-10T17:02:56.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:55 smithi192 bash[23730]: audit 2023-07-10T17:02:55.344459+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:02:56.515 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'sleep 120' 2023-07-10T17:02:56.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:56 smithi116 bash[23740]: audit 2023-07-10T17:02:55.299275+0000 mgr.y (mgr.14144) 309 : audit [DBG] from='client.24694 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:02:56.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:56 smithi116 bash[23740]: cephadm 2023-07-10T17:02:55.300231+0000 mgr.y (mgr.14144) 310 : cephadm [INF] Saving service iscsi.foo spec with placement count:1 2023-07-10T17:02:56.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:56 smithi116 bash[23740]: cephadm 2023-07-10T17:02:55.319536+0000 mgr.y (mgr.14144) 311 : cephadm [INF] Checking pool "foo" exists for service iscsi.foo 2023-07-10T17:02:56.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:56 smithi116 bash[23740]: cephadm 2023-07-10T17:02:55.347275+0000 mgr.y (mgr.14144) 312 : cephadm [INF] Deploying daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:02:56.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:56 smithi116 bash[23740]: cluster 2023-07-10T17:02:55.480487+0000 mgr.y (mgr.14144) 313 : cluster [DBG] pgmap v286: 161 pgs: 161 active+clean; 27 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 311 KiB/s rd, 618 B/s wr, 543 op/s 2023-07-10T17:02:56.958 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:56 smithi116 bash[28351]: audit 2023-07-10T17:02:55.299275+0000 mgr.y (mgr.14144) 309 : audit [DBG] from='client.24694 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:02:56.958 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:56 smithi116 bash[28351]: cephadm 2023-07-10T17:02:55.300231+0000 mgr.y (mgr.14144) 310 : cephadm [INF] Saving service iscsi.foo spec with placement count:1 2023-07-10T17:02:56.958 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:56 smithi116 bash[28351]: cephadm 2023-07-10T17:02:55.319536+0000 mgr.y (mgr.14144) 311 : cephadm [INF] Checking pool "foo" exists for service iscsi.foo 2023-07-10T17:02:56.958 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:56 smithi116 bash[28351]: cephadm 2023-07-10T17:02:55.347275+0000 mgr.y (mgr.14144) 312 : cephadm [INF] Deploying daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:02:56.958 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:56 smithi116 bash[28351]: cluster 2023-07-10T17:02:55.480487+0000 mgr.y (mgr.14144) 313 : cluster [DBG] pgmap v286: 161 pgs: 161 active+clean; 27 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 311 KiB/s rd, 618 B/s wr, 543 op/s 2023-07-10T17:02:57.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:56 smithi192 bash[23730]: audit 2023-07-10T17:02:55.299275+0000 mgr.y (mgr.14144) 309 : audit [DBG] from='client.24694 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:02:57.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:56 smithi192 bash[23730]: cephadm 2023-07-10T17:02:55.300231+0000 mgr.y (mgr.14144) 310 : cephadm [INF] Saving service iscsi.foo spec with placement count:1 2023-07-10T17:02:57.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:56 smithi192 bash[23730]: cephadm 2023-07-10T17:02:55.319536+0000 mgr.y (mgr.14144) 311 : cephadm [INF] Checking pool "foo" exists for service iscsi.foo 2023-07-10T17:02:57.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:56 smithi192 bash[23730]: cephadm 2023-07-10T17:02:55.347275+0000 mgr.y (mgr.14144) 312 : cephadm [INF] Deploying daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:02:57.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:56 smithi192 bash[23730]: cluster 2023-07-10T17:02:55.480487+0000 mgr.y (mgr.14144) 313 : cluster [DBG] pgmap v286: 161 pgs: 161 active+clean; 27 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 311 KiB/s rd, 618 B/s wr, 543 op/s 2023-07-10T17:02:58.838 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:58 smithi116 bash[28351]: cluster 2023-07-10T17:02:57.481623+0000 mgr.y (mgr.14144) 314 : cluster [DBG] pgmap v287: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 288 KiB/s rd, 511 B/s wr, 503 op/s 2023-07-10T17:02:58.839 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:58 smithi116 bash[28351]: cluster 2023-07-10T17:02:57.699305+0000 mon.a (mon.0) 599 : cluster [DBG] mgrmap e15: y(active, since 7m), standbys: x 2023-07-10T17:02:58.839 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:02:58 smithi116 bash[28351]: cluster 2023-07-10T17:02:58.483113+0000 mon.a (mon.0) 600 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2023-07-10T17:02:58.839 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:58 smithi116 bash[23740]: cluster 2023-07-10T17:02:57.481623+0000 mgr.y (mgr.14144) 314 : cluster [DBG] pgmap v287: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 288 KiB/s rd, 511 B/s wr, 503 op/s 2023-07-10T17:02:58.839 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:58 smithi116 bash[23740]: cluster 2023-07-10T17:02:57.699305+0000 mon.a (mon.0) 599 : cluster [DBG] mgrmap e15: y(active, since 7m), standbys: x 2023-07-10T17:02:58.840 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:02:58 smithi116 bash[23740]: cluster 2023-07-10T17:02:58.483113+0000 mon.a (mon.0) 600 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2023-07-10T17:02:59.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:58 smithi192 bash[23730]: cluster 2023-07-10T17:02:57.481623+0000 mgr.y (mgr.14144) 314 : cluster [DBG] pgmap v287: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 288 KiB/s rd, 511 B/s wr, 503 op/s 2023-07-10T17:02:59.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:58 smithi192 bash[23730]: cluster 2023-07-10T17:02:57.699305+0000 mon.a (mon.0) 599 : cluster [DBG] mgrmap e15: y(active, since 7m), standbys: x 2023-07-10T17:02:59.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:02:58 smithi192 bash[23730]: cluster 2023-07-10T17:02:58.483113+0000 mon.a (mon.0) 600 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2023-07-10T17:03:00.403 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:00 smithi116 bash[28351]: audit 2023-07-10T17:02:59.150821+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:00.403 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:00 smithi116 bash[28351]: cluster 2023-07-10T17:02:59.482451+0000 mgr.y (mgr.14144) 315 : cluster [DBG] pgmap v289: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 251 KiB/s rd, 357 B/s wr, 435 op/s 2023-07-10T17:03:00.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:00 smithi116 bash[23740]: audit 2023-07-10T17:02:59.150821+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:00.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:00 smithi116 bash[23740]: cluster 2023-07-10T17:02:59.482451+0000 mgr.y (mgr.14144) 315 : cluster [DBG] pgmap v289: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 251 KiB/s rd, 357 B/s wr, 435 op/s 2023-07-10T17:03:00.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:00 smithi192 bash[23730]: audit 2023-07-10T17:02:59.150821+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:00.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:00 smithi192 bash[23730]: cluster 2023-07-10T17:02:59.482451+0000 mgr.y (mgr.14144) 315 : cluster [DBG] pgmap v289: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 251 KiB/s rd, 357 B/s wr, 435 op/s 2023-07-10T17:03:02.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:02 smithi192 bash[23730]: cluster 2023-07-10T17:03:01.483445+0000 mgr.y (mgr.14144) 316 : cluster [DBG] pgmap v290: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 179 KiB/s rd, 307 B/s wr, 307 op/s 2023-07-10T17:03:02.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:02 smithi192 bash[23730]: audit 2023-07-10T17:03:02.126449+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.116:0/3727901762' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:03:02.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:02 smithi192 bash[23730]: audit 2023-07-10T17:03:02.466619+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.116:0/3196324629' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: access denied 2023-07-10T17:03:02.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:02 smithi116 bash[23740]: cluster 2023-07-10T17:03:01.483445+0000 mgr.y (mgr.14144) 316 : cluster [DBG] pgmap v290: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 179 KiB/s rd, 307 B/s wr, 307 op/s 2023-07-10T17:03:02.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:02 smithi116 bash[23740]: audit 2023-07-10T17:03:02.126449+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.116:0/3727901762' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:03:02.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:02 smithi116 bash[23740]: audit 2023-07-10T17:03:02.466619+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.116:0/3196324629' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: access denied 2023-07-10T17:03:02.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:02 smithi116 bash[28351]: cluster 2023-07-10T17:03:01.483445+0000 mgr.y (mgr.14144) 316 : cluster [DBG] pgmap v290: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 179 KiB/s rd, 307 B/s wr, 307 op/s 2023-07-10T17:03:02.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:02 smithi116 bash[28351]: audit 2023-07-10T17:03:02.126449+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.116:0/3727901762' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:03:02.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:02 smithi116 bash[28351]: audit 2023-07-10T17:03:02.466619+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.116:0/3196324629' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: access denied 2023-07-10T17:03:04.411 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[28351]: audit 2023-07-10T17:03:03.401082+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:04.411 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[28351]: audit 2023-07-10T17:03:03.410008+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:04.412 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[23740]: audit 2023-07-10T17:03:03.401082+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:04.412 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[23740]: audit 2023-07-10T17:03:03.410008+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:04.412 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[23740]: audit 2023-07-10T17:03:03.423956+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:03:04.413 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[23740]: audit 2023-07-10T17:03:03.425404+0000 mgr.y (mgr.14144) 317 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:03:04.413 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[23740]: cephadm 2023-07-10T17:03:03.428655+0000 mgr.y (mgr.14144) 318 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.116:5000 to Dashboard 2023-07-10T17:03:04.413 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[23740]: audit 2023-07-10T17:03:03.429174+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:03:04.413 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[23740]: audit 2023-07-10T17:03:03.430092+0000 mgr.y (mgr.14144) 319 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:03:04.413 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[23740]: audit 2023-07-10T17:03:03.441031+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:04.413 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[23740]: audit 2023-07-10T17:03:03.445520+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi116"}]: dispatch 2023-07-10T17:03:04.414 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[23740]: audit 2023-07-10T17:03:03.446493+0000 mgr.y (mgr.14144) 320 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi116"}]: dispatch 2023-07-10T17:03:04.414 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[23740]: audit 2023-07-10T17:03:03.462140+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:04.414 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[23740]: audit 2023-07-10T17:03:03.474034+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:04.414 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[23740]: cluster 2023-07-10T17:03:03.484357+0000 mgr.y (mgr.14144) 321 : cluster [DBG] pgmap v291: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 102 B/s wr, 138 op/s 2023-07-10T17:03:04.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[28351]: audit 2023-07-10T17:03:03.423956+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:03:04.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[28351]: audit 2023-07-10T17:03:03.425404+0000 mgr.y (mgr.14144) 317 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:03:04.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[28351]: cephadm 2023-07-10T17:03:03.428655+0000 mgr.y (mgr.14144) 318 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.116:5000 to Dashboard 2023-07-10T17:03:04.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[28351]: audit 2023-07-10T17:03:03.429174+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:03:04.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[28351]: audit 2023-07-10T17:03:03.430092+0000 mgr.y (mgr.14144) 319 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:03:04.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[28351]: audit 2023-07-10T17:03:03.441031+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:04.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[28351]: audit 2023-07-10T17:03:03.445520+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi116"}]: dispatch 2023-07-10T17:03:04.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[28351]: audit 2023-07-10T17:03:03.446493+0000 mgr.y (mgr.14144) 320 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi116"}]: dispatch 2023-07-10T17:03:04.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[28351]: audit 2023-07-10T17:03:03.462140+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:04.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[28351]: audit 2023-07-10T17:03:03.474034+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:04.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:04 smithi116 bash[28351]: cluster 2023-07-10T17:03:03.484357+0000 mgr.y (mgr.14144) 321 : cluster [DBG] pgmap v291: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 102 B/s wr, 138 op/s 2023-07-10T17:03:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:04 smithi192 bash[23730]: audit 2023-07-10T17:03:03.401082+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:04 smithi192 bash[23730]: audit 2023-07-10T17:03:03.410008+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:04 smithi192 bash[23730]: audit 2023-07-10T17:03:03.423956+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:03:04.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:04 smithi192 bash[23730]: audit 2023-07-10T17:03:03.425404+0000 mgr.y (mgr.14144) 317 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:03:04.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:04 smithi192 bash[23730]: cephadm 2023-07-10T17:03:03.428655+0000 mgr.y (mgr.14144) 318 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.116:5000 to Dashboard 2023-07-10T17:03:04.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:04 smithi192 bash[23730]: audit 2023-07-10T17:03:03.429174+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:03:04.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:04 smithi192 bash[23730]: audit 2023-07-10T17:03:03.430092+0000 mgr.y (mgr.14144) 319 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:03:04.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:04 smithi192 bash[23730]: audit 2023-07-10T17:03:03.441031+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:04.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:04 smithi192 bash[23730]: audit 2023-07-10T17:03:03.445520+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi116"}]: dispatch 2023-07-10T17:03:04.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:04 smithi192 bash[23730]: audit 2023-07-10T17:03:03.446493+0000 mgr.y (mgr.14144) 320 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi116"}]: dispatch 2023-07-10T17:03:04.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:04 smithi192 bash[23730]: audit 2023-07-10T17:03:03.462140+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:04.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:04 smithi192 bash[23730]: audit 2023-07-10T17:03:03.474034+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:03:04.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:04 smithi192 bash[23730]: cluster 2023-07-10T17:03:03.484357+0000 mgr.y (mgr.14144) 321 : cluster [DBG] pgmap v291: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 102 B/s wr, 138 op/s 2023-07-10T17:03:06.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:06 smithi192 bash[23730]: cluster 2023-07-10T17:03:05.485410+0000 mgr.y (mgr.14144) 322 : cluster [DBG] pgmap v292: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 45 op/s 2023-07-10T17:03:06.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:06 smithi116 bash[28351]: cluster 2023-07-10T17:03:05.485410+0000 mgr.y (mgr.14144) 322 : cluster [DBG] pgmap v292: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 45 op/s 2023-07-10T17:03:06.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:06 smithi116 bash[23740]: cluster 2023-07-10T17:03:05.485410+0000 mgr.y (mgr.14144) 322 : cluster [DBG] pgmap v292: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 45 op/s 2023-07-10T17:03:08.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:08 smithi192 bash[23730]: cluster 2023-07-10T17:03:07.486589+0000 mgr.y (mgr.14144) 323 : cluster [DBG] pgmap v293: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 102 B/s wr, 4 op/s 2023-07-10T17:03:08.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:08 smithi116 bash[23740]: cluster 2023-07-10T17:03:07.486589+0000 mgr.y (mgr.14144) 323 : cluster [DBG] pgmap v293: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 102 B/s wr, 4 op/s 2023-07-10T17:03:08.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:08 smithi116 bash[28351]: cluster 2023-07-10T17:03:07.486589+0000 mgr.y (mgr.14144) 323 : cluster [DBG] pgmap v293: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 102 B/s wr, 4 op/s 2023-07-10T17:03:10.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:10 smithi192 bash[23730]: cluster 2023-07-10T17:03:09.487949+0000 mgr.y (mgr.14144) 324 : cluster [DBG] pgmap v294: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 92 B/s rd, 92 B/s wr, 0 op/s 2023-07-10T17:03:10.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:10 smithi116 bash[23740]: cluster 2023-07-10T17:03:09.487949+0000 mgr.y (mgr.14144) 324 : cluster [DBG] pgmap v294: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 92 B/s rd, 92 B/s wr, 0 op/s 2023-07-10T17:03:10.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:10 smithi116 bash[28351]: cluster 2023-07-10T17:03:09.487949+0000 mgr.y (mgr.14144) 324 : cluster [DBG] pgmap v294: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 92 B/s rd, 92 B/s wr, 0 op/s 2023-07-10T17:03:12.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:12 smithi192 bash[23730]: cluster 2023-07-10T17:03:11.489076+0000 mgr.y (mgr.14144) 325 : cluster [DBG] pgmap v295: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 0 op/s 2023-07-10T17:03:12.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:12 smithi116 bash[28351]: cluster 2023-07-10T17:03:11.489076+0000 mgr.y (mgr.14144) 325 : cluster [DBG] pgmap v295: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 0 op/s 2023-07-10T17:03:12.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:12 smithi116 bash[23740]: cluster 2023-07-10T17:03:11.489076+0000 mgr.y (mgr.14144) 325 : cluster [DBG] pgmap v295: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 0 op/s 2023-07-10T17:03:14.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:14 smithi192 bash[23730]: cluster 2023-07-10T17:03:13.490298+0000 mgr.y (mgr.14144) 326 : cluster [DBG] pgmap v296: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 0 op/s 2023-07-10T17:03:14.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:14 smithi116 bash[23740]: cluster 2023-07-10T17:03:13.490298+0000 mgr.y (mgr.14144) 326 : cluster [DBG] pgmap v296: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 0 op/s 2023-07-10T17:03:14.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:14 smithi116 bash[28351]: cluster 2023-07-10T17:03:13.490298+0000 mgr.y (mgr.14144) 326 : cluster [DBG] pgmap v296: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 0 op/s 2023-07-10T17:03:16.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:16 smithi192 bash[23730]: cluster 2023-07-10T17:03:15.491127+0000 mgr.y (mgr.14144) 327 : cluster [DBG] pgmap v297: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 0 op/s 2023-07-10T17:03:16.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:16 smithi116 bash[23740]: cluster 2023-07-10T17:03:15.491127+0000 mgr.y (mgr.14144) 327 : cluster [DBG] pgmap v297: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 0 op/s 2023-07-10T17:03:16.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:16 smithi116 bash[28351]: cluster 2023-07-10T17:03:15.491127+0000 mgr.y (mgr.14144) 327 : cluster [DBG] pgmap v297: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 0 op/s 2023-07-10T17:03:17.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:17 smithi116 bash[23740]: audit 2023-07-10T17:03:16.811463+0000 mon.a (mon.0) 610 : audit [DBG] from='client.? 172.21.15.116:0/1814751039' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:03:17.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:17 smithi116 bash[23740]: audit 2023-07-10T17:03:17.141079+0000 mon.a (mon.0) 611 : audit [INF] from='client.? 172.21.15.116:0/2760596321' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: access denied 2023-07-10T17:03:17.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:17 smithi116 bash[28351]: audit 2023-07-10T17:03:16.811463+0000 mon.a (mon.0) 610 : audit [DBG] from='client.? 172.21.15.116:0/1814751039' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:03:17.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:17 smithi116 bash[28351]: audit 2023-07-10T17:03:17.141079+0000 mon.a (mon.0) 611 : audit [INF] from='client.? 172.21.15.116:0/2760596321' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: access denied 2023-07-10T17:03:17.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:17 smithi192 bash[23730]: audit 2023-07-10T17:03:16.811463+0000 mon.a (mon.0) 610 : audit [DBG] from='client.? 172.21.15.116:0/1814751039' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:03:17.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:17 smithi192 bash[23730]: audit 2023-07-10T17:03:17.141079+0000 mon.a (mon.0) 611 : audit [INF] from='client.? 172.21.15.116:0/2760596321' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: access denied 2023-07-10T17:03:18.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:18 smithi192 bash[23730]: cluster 2023-07-10T17:03:17.492204+0000 mgr.y (mgr.14144) 328 : cluster [DBG] pgmap v298: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 0 op/s 2023-07-10T17:03:18.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:18 smithi116 bash[23740]: cluster 2023-07-10T17:03:17.492204+0000 mgr.y (mgr.14144) 328 : cluster [DBG] pgmap v298: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 0 op/s 2023-07-10T17:03:18.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:18 smithi116 bash[28351]: cluster 2023-07-10T17:03:17.492204+0000 mgr.y (mgr.14144) 328 : cluster [DBG] pgmap v298: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 0 op/s 2023-07-10T17:03:20.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:20 smithi192 bash[23730]: cluster 2023-07-10T17:03:19.493494+0000 mgr.y (mgr.14144) 329 : cluster [DBG] pgmap v299: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:03:20.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:20 smithi192 bash[23730]: audit 2023-07-10T17:03:19.613983+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:03:20.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:20 smithi192 bash[23730]: audit 2023-07-10T17:03:19.616804+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:03:20.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:20 smithi192 bash[23730]: audit 2023-07-10T17:03:19.682213+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2023-07-10T17:03:20.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:20 smithi192 bash[23730]: audit 2023-07-10T17:03:19.682989+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2023-07-10T17:03:20.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:20 smithi192 bash[23730]: audit 2023-07-10T17:03:19.683755+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2023-07-10T17:03:20.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:20 smithi116 bash[23740]: cluster 2023-07-10T17:03:19.493494+0000 mgr.y (mgr.14144) 329 : cluster [DBG] pgmap v299: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:03:20.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:20 smithi116 bash[23740]: audit 2023-07-10T17:03:19.613983+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:03:20.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:20 smithi116 bash[23740]: audit 2023-07-10T17:03:19.616804+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:03:20.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:20 smithi116 bash[23740]: audit 2023-07-10T17:03:19.682213+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2023-07-10T17:03:20.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:20 smithi116 bash[23740]: audit 2023-07-10T17:03:19.682989+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2023-07-10T17:03:20.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:20 smithi116 bash[23740]: audit 2023-07-10T17:03:19.683755+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2023-07-10T17:03:20.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:20 smithi116 bash[28351]: cluster 2023-07-10T17:03:19.493494+0000 mgr.y (mgr.14144) 329 : cluster [DBG] pgmap v299: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:03:20.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:20 smithi116 bash[28351]: audit 2023-07-10T17:03:19.613983+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:03:20.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:20 smithi116 bash[28351]: audit 2023-07-10T17:03:19.616804+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:03:20.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:20 smithi116 bash[28351]: audit 2023-07-10T17:03:19.682213+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2023-07-10T17:03:20.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:20 smithi116 bash[28351]: audit 2023-07-10T17:03:19.682989+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2023-07-10T17:03:20.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:20 smithi116 bash[28351]: audit 2023-07-10T17:03:19.683755+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2023-07-10T17:03:21.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:21 smithi192 bash[23730]: audit 2023-07-10T17:03:20.578474+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2023-07-10T17:03:21.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:21 smithi192 bash[23730]: audit 2023-07-10T17:03:20.578621+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2023-07-10T17:03:21.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:21 smithi192 bash[23730]: audit 2023-07-10T17:03:20.578751+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2023-07-10T17:03:21.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:21 smithi192 bash[23730]: cluster 2023-07-10T17:03:20.578870+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2023-07-10T17:03:21.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:21 smithi192 bash[23730]: cluster 2023-07-10T17:03:21.581068+0000 mon.a (mon.0) 621 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2023-07-10T17:03:21.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:21 smithi116 bash[28351]: audit 2023-07-10T17:03:20.578474+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2023-07-10T17:03:21.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:21 smithi116 bash[28351]: audit 2023-07-10T17:03:20.578621+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2023-07-10T17:03:21.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:21 smithi116 bash[28351]: audit 2023-07-10T17:03:20.578751+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2023-07-10T17:03:21.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:21 smithi116 bash[28351]: cluster 2023-07-10T17:03:20.578870+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2023-07-10T17:03:21.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:21 smithi116 bash[28351]: cluster 2023-07-10T17:03:21.581068+0000 mon.a (mon.0) 621 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2023-07-10T17:03:21.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:21 smithi116 bash[23740]: audit 2023-07-10T17:03:20.578474+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2023-07-10T17:03:21.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:21 smithi116 bash[23740]: audit 2023-07-10T17:03:20.578621+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2023-07-10T17:03:21.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:21 smithi116 bash[23740]: audit 2023-07-10T17:03:20.578751+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2023-07-10T17:03:21.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:21 smithi116 bash[23740]: cluster 2023-07-10T17:03:20.578870+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2023-07-10T17:03:21.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:21 smithi116 bash[23740]: cluster 2023-07-10T17:03:21.581068+0000 mon.a (mon.0) 621 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2023-07-10T17:03:22.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:22 smithi192 bash[23730]: cluster 2023-07-10T17:03:21.494500+0000 mgr.y (mgr.14144) 330 : cluster [DBG] pgmap v301: 161 pgs: 2 peering, 159 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-07-10T17:03:22.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:22 smithi116 bash[23740]: cluster 2023-07-10T17:03:21.494500+0000 mgr.y (mgr.14144) 330 : cluster [DBG] pgmap v301: 161 pgs: 2 peering, 159 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-07-10T17:03:22.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:22 smithi116 bash[28351]: cluster 2023-07-10T17:03:21.494500+0000 mgr.y (mgr.14144) 330 : cluster [DBG] pgmap v301: 161 pgs: 2 peering, 159 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-07-10T17:03:24.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:24 smithi192 bash[23730]: cluster 2023-07-10T17:03:23.495810+0000 mgr.y (mgr.14144) 331 : cluster [DBG] pgmap v303: 161 pgs: 2 peering, 159 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2023-07-10T17:03:24.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:24 smithi116 bash[23740]: cluster 2023-07-10T17:03:23.495810+0000 mgr.y (mgr.14144) 331 : cluster [DBG] pgmap v303: 161 pgs: 2 peering, 159 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2023-07-10T17:03:24.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:24 smithi116 bash[28351]: cluster 2023-07-10T17:03:23.495810+0000 mgr.y (mgr.14144) 331 : cluster [DBG] pgmap v303: 161 pgs: 2 peering, 159 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2023-07-10T17:03:26.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:26 smithi192 bash[23730]: cluster 2023-07-10T17:03:25.496974+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v304: 161 pgs: 2 peering, 159 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2023-07-10T17:03:26.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:26 smithi116 bash[23740]: cluster 2023-07-10T17:03:25.496974+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v304: 161 pgs: 2 peering, 159 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2023-07-10T17:03:26.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:26 smithi116 bash[28351]: cluster 2023-07-10T17:03:25.496974+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v304: 161 pgs: 2 peering, 159 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2023-07-10T17:03:28.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:28 smithi192 bash[23730]: cluster 2023-07-10T17:03:27.498155+0000 mgr.y (mgr.14144) 333 : cluster [DBG] pgmap v305: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2023-07-10T17:03:28.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:28 smithi116 bash[28351]: cluster 2023-07-10T17:03:27.498155+0000 mgr.y (mgr.14144) 333 : cluster [DBG] pgmap v305: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2023-07-10T17:03:28.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:28 smithi116 bash[23740]: cluster 2023-07-10T17:03:27.498155+0000 mgr.y (mgr.14144) 333 : cluster [DBG] pgmap v305: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2023-07-10T17:03:30.642 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:30 smithi116 bash[28351]: cluster 2023-07-10T17:03:29.499509+0000 mgr.y (mgr.14144) 334 : cluster [DBG] pgmap v306: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-07-10T17:03:30.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:30 smithi116 bash[23740]: cluster 2023-07-10T17:03:29.499509+0000 mgr.y (mgr.14144) 334 : cluster [DBG] pgmap v306: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-07-10T17:03:30.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:30 smithi192 bash[23730]: cluster 2023-07-10T17:03:29.499509+0000 mgr.y (mgr.14144) 334 : cluster [DBG] pgmap v306: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-07-10T17:03:31.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:31 smithi116 bash[23740]: audit 2023-07-10T17:03:31.040874+0000 mon.c (mon.1) 31 : audit [DBG] from='client.? 172.21.15.116:0/2456559958' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:03:31.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:31 smithi116 bash[23740]: audit 2023-07-10T17:03:31.369915+0000 mon.c (mon.1) 32 : audit [INF] from='client.? 172.21.15.116:0/2109156200' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: access denied 2023-07-10T17:03:31.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:31 smithi116 bash[28351]: audit 2023-07-10T17:03:31.040874+0000 mon.c (mon.1) 31 : audit [DBG] from='client.? 172.21.15.116:0/2456559958' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:03:31.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:31 smithi116 bash[28351]: audit 2023-07-10T17:03:31.369915+0000 mon.c (mon.1) 32 : audit [INF] from='client.? 172.21.15.116:0/2109156200' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: access denied 2023-07-10T17:03:31.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:31 smithi192 bash[23730]: audit 2023-07-10T17:03:31.040874+0000 mon.c (mon.1) 31 : audit [DBG] from='client.? 172.21.15.116:0/2456559958' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:03:31.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:31 smithi192 bash[23730]: audit 2023-07-10T17:03:31.369915+0000 mon.c (mon.1) 32 : audit [INF] from='client.? 172.21.15.116:0/2109156200' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: access denied 2023-07-10T17:03:32.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:32 smithi192 bash[23730]: cluster 2023-07-10T17:03:31.500586+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v307: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-07-10T17:03:32.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:32 smithi116 bash[23740]: cluster 2023-07-10T17:03:31.500586+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v307: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-07-10T17:03:32.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:32 smithi116 bash[28351]: cluster 2023-07-10T17:03:31.500586+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v307: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-07-10T17:03:34.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:34 smithi192 bash[23730]: cluster 2023-07-10T17:03:33.501811+0000 mgr.y (mgr.14144) 336 : cluster [DBG] pgmap v308: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:34.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:34 smithi116 bash[23740]: cluster 2023-07-10T17:03:33.501811+0000 mgr.y (mgr.14144) 336 : cluster [DBG] pgmap v308: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:34.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:34 smithi116 bash[28351]: cluster 2023-07-10T17:03:33.501811+0000 mgr.y (mgr.14144) 336 : cluster [DBG] pgmap v308: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:36.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:36 smithi192 bash[23730]: cluster 2023-07-10T17:03:35.502869+0000 mgr.y (mgr.14144) 337 : cluster [DBG] pgmap v309: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:36.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:36 smithi116 bash[28351]: cluster 2023-07-10T17:03:35.502869+0000 mgr.y (mgr.14144) 337 : cluster [DBG] pgmap v309: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:36.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:36 smithi116 bash[23740]: cluster 2023-07-10T17:03:35.502869+0000 mgr.y (mgr.14144) 337 : cluster [DBG] pgmap v309: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:38.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:38 smithi192 bash[23730]: cluster 2023-07-10T17:03:37.504011+0000 mgr.y (mgr.14144) 338 : cluster [DBG] pgmap v310: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:38.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:38 smithi116 bash[23740]: cluster 2023-07-10T17:03:37.504011+0000 mgr.y (mgr.14144) 338 : cluster [DBG] pgmap v310: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:38.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:38 smithi116 bash[28351]: cluster 2023-07-10T17:03:37.504011+0000 mgr.y (mgr.14144) 338 : cluster [DBG] pgmap v310: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:40.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:40 smithi192 bash[23730]: cluster 2023-07-10T17:03:39.505305+0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v311: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:40.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:40 smithi116 bash[23740]: cluster 2023-07-10T17:03:39.505305+0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v311: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:40.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:40 smithi116 bash[28351]: cluster 2023-07-10T17:03:39.505305+0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v311: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:42.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:42 smithi192 bash[23730]: cluster 2023-07-10T17:03:41.506450+0000 mgr.y (mgr.14144) 340 : cluster [DBG] pgmap v312: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:42.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:42 smithi116 bash[23740]: cluster 2023-07-10T17:03:41.506450+0000 mgr.y (mgr.14144) 340 : cluster [DBG] pgmap v312: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:42.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:42 smithi116 bash[28351]: cluster 2023-07-10T17:03:41.506450+0000 mgr.y (mgr.14144) 340 : cluster [DBG] pgmap v312: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:44.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:44 smithi192 bash[23730]: cluster 2023-07-10T17:03:43.507812+0000 mgr.y (mgr.14144) 341 : cluster [DBG] pgmap v313: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:03:44.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:44 smithi116 bash[23740]: cluster 2023-07-10T17:03:43.507812+0000 mgr.y (mgr.14144) 341 : cluster [DBG] pgmap v313: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:03:44.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:44 smithi116 bash[28351]: cluster 2023-07-10T17:03:43.507812+0000 mgr.y (mgr.14144) 341 : cluster [DBG] pgmap v313: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:03:45.882 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:45 smithi116 bash[28351]: audit 2023-07-10T17:03:45.304076+0000 mon.a (mon.0) 622 : audit [DBG] from='client.? 172.21.15.116:0/340813155' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:03:45.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:45 smithi116 bash[23740]: audit 2023-07-10T17:03:45.304076+0000 mon.a (mon.0) 622 : audit [DBG] from='client.? 172.21.15.116:0/340813155' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:03:45.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:45 smithi192 bash[23730]: audit 2023-07-10T17:03:45.304076+0000 mon.a (mon.0) 622 : audit [DBG] from='client.? 172.21.15.116:0/340813155' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:03:46.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:46 smithi116 bash[28351]: cluster 2023-07-10T17:03:45.508866+0000 mgr.y (mgr.14144) 342 : cluster [DBG] pgmap v314: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:03:46.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:46 smithi116 bash[28351]: audit 2023-07-10T17:03:45.649692+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.116:0/764742741' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: access denied 2023-07-10T17:03:46.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:46 smithi116 bash[23740]: cluster 2023-07-10T17:03:45.508866+0000 mgr.y (mgr.14144) 342 : cluster [DBG] pgmap v314: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:03:46.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:46 smithi116 bash[23740]: audit 2023-07-10T17:03:45.649692+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.116:0/764742741' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: access denied 2023-07-10T17:03:46.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:46 smithi192 bash[23730]: cluster 2023-07-10T17:03:45.508866+0000 mgr.y (mgr.14144) 342 : cluster [DBG] pgmap v314: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:03:46.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:46 smithi192 bash[23730]: audit 2023-07-10T17:03:45.649692+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.116:0/764742741' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: access denied 2023-07-10T17:03:48.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:48 smithi192 bash[23730]: cluster 2023-07-10T17:03:47.509896+0000 mgr.y (mgr.14144) 343 : cluster [DBG] pgmap v315: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:48.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:48 smithi116 bash[23740]: cluster 2023-07-10T17:03:47.509896+0000 mgr.y (mgr.14144) 343 : cluster [DBG] pgmap v315: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:48.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:48 smithi116 bash[28351]: cluster 2023-07-10T17:03:47.509896+0000 mgr.y (mgr.14144) 343 : cluster [DBG] pgmap v315: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:50.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:50 smithi192 bash[23730]: cluster 2023-07-10T17:03:49.511199+0000 mgr.y (mgr.14144) 344 : cluster [DBG] pgmap v316: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:50.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:50 smithi116 bash[28351]: cluster 2023-07-10T17:03:49.511199+0000 mgr.y (mgr.14144) 344 : cluster [DBG] pgmap v316: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:50.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:50 smithi116 bash[23740]: cluster 2023-07-10T17:03:49.511199+0000 mgr.y (mgr.14144) 344 : cluster [DBG] pgmap v316: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:52.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:52 smithi192 bash[23730]: cluster 2023-07-10T17:03:51.512391+0000 mgr.y (mgr.14144) 345 : cluster [DBG] pgmap v317: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:52.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:52 smithi116 bash[23740]: cluster 2023-07-10T17:03:51.512391+0000 mgr.y (mgr.14144) 345 : cluster [DBG] pgmap v317: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:52.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:52 smithi116 bash[28351]: cluster 2023-07-10T17:03:51.512391+0000 mgr.y (mgr.14144) 345 : cluster [DBG] pgmap v317: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:54.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:54 smithi192 bash[23730]: cluster 2023-07-10T17:03:53.513780+0000 mgr.y (mgr.14144) 346 : cluster [DBG] pgmap v318: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:54.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:54 smithi116 bash[28351]: cluster 2023-07-10T17:03:53.513780+0000 mgr.y (mgr.14144) 346 : cluster [DBG] pgmap v318: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:54.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:54 smithi116 bash[23740]: cluster 2023-07-10T17:03:53.513780+0000 mgr.y (mgr.14144) 346 : cluster [DBG] pgmap v318: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:56.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:56 smithi192 bash[23730]: cluster 2023-07-10T17:03:55.514876+0000 mgr.y (mgr.14144) 347 : cluster [DBG] pgmap v319: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:56.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:56 smithi116 bash[23740]: cluster 2023-07-10T17:03:55.514876+0000 mgr.y (mgr.14144) 347 : cluster [DBG] pgmap v319: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:56.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:56 smithi116 bash[28351]: cluster 2023-07-10T17:03:55.514876+0000 mgr.y (mgr.14144) 347 : cluster [DBG] pgmap v319: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:58.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:58 smithi192 bash[23730]: cluster 2023-07-10T17:03:57.516001+0000 mgr.y (mgr.14144) 348 : cluster [DBG] pgmap v320: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:58.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:58 smithi116 bash[28351]: cluster 2023-07-10T17:03:57.516001+0000 mgr.y (mgr.14144) 348 : cluster [DBG] pgmap v320: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:58.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:58 smithi116 bash[23740]: cluster 2023-07-10T17:03:57.516001+0000 mgr.y (mgr.14144) 348 : cluster [DBG] pgmap v320: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:03:59.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:59 smithi192 bash[23730]: audit 2023-07-10T17:03:59.179520+0000 mon.a (mon.0) 623 : audit [DBG] from='client.? 172.21.15.116:0/1227756522' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:03:59.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:03:59 smithi192 bash[23730]: audit 2023-07-10T17:03:59.516326+0000 mon.c (mon.1) 34 : audit [INF] from='client.? 172.21.15.116:0/3820490624' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: access denied 2023-07-10T17:03:59.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:59 smithi116 bash[23740]: audit 2023-07-10T17:03:59.179520+0000 mon.a (mon.0) 623 : audit [DBG] from='client.? 172.21.15.116:0/1227756522' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:03:59.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:03:59 smithi116 bash[23740]: audit 2023-07-10T17:03:59.516326+0000 mon.c (mon.1) 34 : audit [INF] from='client.? 172.21.15.116:0/3820490624' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: access denied 2023-07-10T17:03:59.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:59 smithi116 bash[28351]: audit 2023-07-10T17:03:59.179520+0000 mon.a (mon.0) 623 : audit [DBG] from='client.? 172.21.15.116:0/1227756522' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:03:59.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:03:59 smithi116 bash[28351]: audit 2023-07-10T17:03:59.516326+0000 mon.c (mon.1) 34 : audit [INF] from='client.? 172.21.15.116:0/3820490624' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: access denied 2023-07-10T17:04:00.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:00 smithi192 bash[23730]: cluster 2023-07-10T17:03:59.517516+0000 mgr.y (mgr.14144) 349 : cluster [DBG] pgmap v321: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:00.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:00 smithi116 bash[23740]: cluster 2023-07-10T17:03:59.517516+0000 mgr.y (mgr.14144) 349 : cluster [DBG] pgmap v321: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:00.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:00 smithi116 bash[28351]: cluster 2023-07-10T17:03:59.517516+0000 mgr.y (mgr.14144) 349 : cluster [DBG] pgmap v321: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:02.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:02 smithi192 bash[23730]: cluster 2023-07-10T17:04:01.518760+0000 mgr.y (mgr.14144) 350 : cluster [DBG] pgmap v322: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:02.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:02 smithi116 bash[23740]: cluster 2023-07-10T17:04:01.518760+0000 mgr.y (mgr.14144) 350 : cluster [DBG] pgmap v322: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:02.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:02 smithi116 bash[28351]: cluster 2023-07-10T17:04:01.518760+0000 mgr.y (mgr.14144) 350 : cluster [DBG] pgmap v322: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:05.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:04 smithi192 bash[23730]: cluster 2023-07-10T17:04:03.519874+0000 mgr.y (mgr.14144) 351 : cluster [DBG] pgmap v323: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:05.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:04 smithi192 bash[23730]: audit 2023-07-10T17:04:03.734839+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:04:05.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:04 smithi116 bash[28351]: cluster 2023-07-10T17:04:03.519874+0000 mgr.y (mgr.14144) 351 : cluster [DBG] pgmap v323: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:05.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:04 smithi116 bash[28351]: audit 2023-07-10T17:04:03.734839+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:04:05.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:04 smithi116 bash[23740]: cluster 2023-07-10T17:04:03.519874+0000 mgr.y (mgr.14144) 351 : cluster [DBG] pgmap v323: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:05.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:04 smithi116 bash[23740]: audit 2023-07-10T17:04:03.734839+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:04:07.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:06 smithi192 bash[23730]: cluster 2023-07-10T17:04:05.520933+0000 mgr.y (mgr.14144) 352 : cluster [DBG] pgmap v324: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:07.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:06 smithi116 bash[28351]: cluster 2023-07-10T17:04:05.520933+0000 mgr.y (mgr.14144) 352 : cluster [DBG] pgmap v324: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:07.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:06 smithi116 bash[23740]: cluster 2023-07-10T17:04:05.520933+0000 mgr.y (mgr.14144) 352 : cluster [DBG] pgmap v324: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:09.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:08 smithi192 bash[23730]: cluster 2023-07-10T17:04:07.521693+0000 mgr.y (mgr.14144) 353 : cluster [DBG] pgmap v325: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:09.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:08 smithi116 bash[28351]: cluster 2023-07-10T17:04:07.521693+0000 mgr.y (mgr.14144) 353 : cluster [DBG] pgmap v325: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:09.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:08 smithi116 bash[23740]: cluster 2023-07-10T17:04:07.521693+0000 mgr.y (mgr.14144) 353 : cluster [DBG] pgmap v325: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:10.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:09 smithi192 bash[23730]: cluster 2023-07-10T17:04:09.522926+0000 mgr.y (mgr.14144) 354 : cluster [DBG] pgmap v326: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:10.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:09 smithi116 bash[23740]: cluster 2023-07-10T17:04:09.522926+0000 mgr.y (mgr.14144) 354 : cluster [DBG] pgmap v326: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:10.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:09 smithi116 bash[28351]: cluster 2023-07-10T17:04:09.522926+0000 mgr.y (mgr.14144) 354 : cluster [DBG] pgmap v326: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:12.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:12 smithi192 bash[23730]: cluster 2023-07-10T17:04:11.523937+0000 mgr.y (mgr.14144) 355 : cluster [DBG] pgmap v327: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:12.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:12 smithi116 bash[23740]: cluster 2023-07-10T17:04:11.523937+0000 mgr.y (mgr.14144) 355 : cluster [DBG] pgmap v327: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:12.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:12 smithi116 bash[28351]: cluster 2023-07-10T17:04:11.523937+0000 mgr.y (mgr.14144) 355 : cluster [DBG] pgmap v327: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:04:14.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:14 smithi192 bash[23730]: cluster 2023-07-10T17:04:13.525192+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v328: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:14.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:14 smithi116 bash[23740]: cluster 2023-07-10T17:04:13.525192+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v328: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:14.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:14 smithi116 bash[28351]: cluster 2023-07-10T17:04:13.525192+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v328: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:16.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:16 smithi192 bash[23730]: cluster 2023-07-10T17:04:15.526096+0000 mgr.y (mgr.14144) 357 : cluster [DBG] pgmap v329: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:16.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:16 smithi116 bash[23740]: cluster 2023-07-10T17:04:15.526096+0000 mgr.y (mgr.14144) 357 : cluster [DBG] pgmap v329: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:16.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:16 smithi116 bash[28351]: cluster 2023-07-10T17:04:15.526096+0000 mgr.y (mgr.14144) 357 : cluster [DBG] pgmap v329: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:18.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:18 smithi192 bash[23730]: cluster 2023-07-10T17:04:17.527240+0000 mgr.y (mgr.14144) 358 : cluster [DBG] pgmap v330: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:18.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:18 smithi116 bash[23740]: cluster 2023-07-10T17:04:17.527240+0000 mgr.y (mgr.14144) 358 : cluster [DBG] pgmap v330: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:18.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:18 smithi116 bash[28351]: cluster 2023-07-10T17:04:17.527240+0000 mgr.y (mgr.14144) 358 : cluster [DBG] pgmap v330: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:20.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:20 smithi192 bash[23730]: cluster 2023-07-10T17:04:19.528524+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v331: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:20.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:20 smithi192 bash[23730]: audit 2023-07-10T17:04:19.626217+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:04:20.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:20 smithi192 bash[23730]: audit 2023-07-10T17:04:19.626512+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:04:20.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:20 smithi116 bash[23740]: cluster 2023-07-10T17:04:19.528524+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v331: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:20.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:20 smithi116 bash[23740]: audit 2023-07-10T17:04:19.626217+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:04:20.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:20 smithi116 bash[23740]: audit 2023-07-10T17:04:19.626512+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:04:20.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:20 smithi116 bash[28351]: cluster 2023-07-10T17:04:19.528524+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v331: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:20.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:20 smithi116 bash[28351]: audit 2023-07-10T17:04:19.626217+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:04:20.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:20 smithi116 bash[28351]: audit 2023-07-10T17:04:19.626512+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:04:22.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:22 smithi192 bash[23730]: cluster 2023-07-10T17:04:21.529582+0000 mgr.y (mgr.14144) 360 : cluster [DBG] pgmap v332: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:22.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:22 smithi116 bash[23740]: cluster 2023-07-10T17:04:21.529582+0000 mgr.y (mgr.14144) 360 : cluster [DBG] pgmap v332: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:22.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:22 smithi116 bash[28351]: cluster 2023-07-10T17:04:21.529582+0000 mgr.y (mgr.14144) 360 : cluster [DBG] pgmap v332: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:24.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:24 smithi192 bash[23730]: cluster 2023-07-10T17:04:23.530419+0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v333: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:24.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:24 smithi116 bash[23740]: cluster 2023-07-10T17:04:23.530419+0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v333: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:24.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:24 smithi116 bash[28351]: cluster 2023-07-10T17:04:23.530419+0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v333: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:26.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:26 smithi192 bash[23730]: cluster 2023-07-10T17:04:25.531558+0000 mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v334: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:26.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:26 smithi116 bash[23740]: cluster 2023-07-10T17:04:25.531558+0000 mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v334: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:26.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:26 smithi116 bash[28351]: cluster 2023-07-10T17:04:25.531558+0000 mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v334: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:28.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:28 smithi192 bash[23730]: cluster 2023-07-10T17:04:27.532852+0000 mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v335: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:28.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:28 smithi116 bash[23740]: cluster 2023-07-10T17:04:27.532852+0000 mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v335: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:28.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:28 smithi116 bash[28351]: cluster 2023-07-10T17:04:27.532852+0000 mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v335: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:30.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:30 smithi192 bash[23730]: cluster 2023-07-10T17:04:29.534275+0000 mgr.y (mgr.14144) 364 : cluster [DBG] pgmap v336: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:30.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:30 smithi116 bash[23740]: cluster 2023-07-10T17:04:29.534275+0000 mgr.y (mgr.14144) 364 : cluster [DBG] pgmap v336: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:30.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:30 smithi116 bash[28351]: cluster 2023-07-10T17:04:29.534275+0000 mgr.y (mgr.14144) 364 : cluster [DBG] pgmap v336: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:32.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:32 smithi192 bash[23730]: cluster 2023-07-10T17:04:31.535307+0000 mgr.y (mgr.14144) 365 : cluster [DBG] pgmap v337: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:32.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:32 smithi116 bash[23740]: cluster 2023-07-10T17:04:31.535307+0000 mgr.y (mgr.14144) 365 : cluster [DBG] pgmap v337: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:32.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:32 smithi116 bash[28351]: cluster 2023-07-10T17:04:31.535307+0000 mgr.y (mgr.14144) 365 : cluster [DBG] pgmap v337: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:34.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:34 smithi192 bash[23730]: cluster 2023-07-10T17:04:33.536552+0000 mgr.y (mgr.14144) 366 : cluster [DBG] pgmap v338: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:34.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:34 smithi116 bash[23740]: cluster 2023-07-10T17:04:33.536552+0000 mgr.y (mgr.14144) 366 : cluster [DBG] pgmap v338: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:34.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:34 smithi116 bash[28351]: cluster 2023-07-10T17:04:33.536552+0000 mgr.y (mgr.14144) 366 : cluster [DBG] pgmap v338: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:36.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:36 smithi192 bash[23730]: cluster 2023-07-10T17:04:35.537633+0000 mgr.y (mgr.14144) 367 : cluster [DBG] pgmap v339: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:36.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:36 smithi116 bash[23740]: cluster 2023-07-10T17:04:35.537633+0000 mgr.y (mgr.14144) 367 : cluster [DBG] pgmap v339: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:36.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:36 smithi116 bash[28351]: cluster 2023-07-10T17:04:35.537633+0000 mgr.y (mgr.14144) 367 : cluster [DBG] pgmap v339: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:38.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:38 smithi192 bash[23730]: cluster 2023-07-10T17:04:37.538829+0000 mgr.y (mgr.14144) 368 : cluster [DBG] pgmap v340: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:38.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:38 smithi116 bash[23740]: cluster 2023-07-10T17:04:37.538829+0000 mgr.y (mgr.14144) 368 : cluster [DBG] pgmap v340: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:38.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:38 smithi116 bash[28351]: cluster 2023-07-10T17:04:37.538829+0000 mgr.y (mgr.14144) 368 : cluster [DBG] pgmap v340: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:40.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:40 smithi192 bash[23730]: cluster 2023-07-10T17:04:39.540078+0000 mgr.y (mgr.14144) 369 : cluster [DBG] pgmap v341: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:40.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:40 smithi116 bash[28351]: cluster 2023-07-10T17:04:39.540078+0000 mgr.y (mgr.14144) 369 : cluster [DBG] pgmap v341: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:40.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:40 smithi116 bash[23740]: cluster 2023-07-10T17:04:39.540078+0000 mgr.y (mgr.14144) 369 : cluster [DBG] pgmap v341: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:42.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:42 smithi192 bash[23730]: cluster 2023-07-10T17:04:41.541090+0000 mgr.y (mgr.14144) 370 : cluster [DBG] pgmap v342: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:42.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:42 smithi116 bash[28351]: cluster 2023-07-10T17:04:41.541090+0000 mgr.y (mgr.14144) 370 : cluster [DBG] pgmap v342: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:42.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:42 smithi116 bash[23740]: cluster 2023-07-10T17:04:41.541090+0000 mgr.y (mgr.14144) 370 : cluster [DBG] pgmap v342: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:44.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:44 smithi192 bash[23730]: cluster 2023-07-10T17:04:43.542380+0000 mgr.y (mgr.14144) 371 : cluster [DBG] pgmap v343: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:44.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:44 smithi116 bash[23740]: cluster 2023-07-10T17:04:43.542380+0000 mgr.y (mgr.14144) 371 : cluster [DBG] pgmap v343: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:44.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:44 smithi116 bash[28351]: cluster 2023-07-10T17:04:43.542380+0000 mgr.y (mgr.14144) 371 : cluster [DBG] pgmap v343: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:46.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:46 smithi192 bash[23730]: cluster 2023-07-10T17:04:45.543475+0000 mgr.y (mgr.14144) 372 : cluster [DBG] pgmap v344: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:46.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:46 smithi116 bash[23740]: cluster 2023-07-10T17:04:45.543475+0000 mgr.y (mgr.14144) 372 : cluster [DBG] pgmap v344: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:46.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:46 smithi116 bash[28351]: cluster 2023-07-10T17:04:45.543475+0000 mgr.y (mgr.14144) 372 : cluster [DBG] pgmap v344: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:48.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:48 smithi192 bash[23730]: cluster 2023-07-10T17:04:47.544617+0000 mgr.y (mgr.14144) 373 : cluster [DBG] pgmap v345: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:48.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:48 smithi116 bash[28351]: cluster 2023-07-10T17:04:47.544617+0000 mgr.y (mgr.14144) 373 : cluster [DBG] pgmap v345: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:48.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:48 smithi116 bash[23740]: cluster 2023-07-10T17:04:47.544617+0000 mgr.y (mgr.14144) 373 : cluster [DBG] pgmap v345: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:50.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:50 smithi192 bash[23730]: cluster 2023-07-10T17:04:49.545882+0000 mgr.y (mgr.14144) 374 : cluster [DBG] pgmap v346: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:50.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:50 smithi116 bash[23740]: cluster 2023-07-10T17:04:49.545882+0000 mgr.y (mgr.14144) 374 : cluster [DBG] pgmap v346: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:50.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:50 smithi116 bash[28351]: cluster 2023-07-10T17:04:49.545882+0000 mgr.y (mgr.14144) 374 : cluster [DBG] pgmap v346: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:52.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:52 smithi192 bash[23730]: cluster 2023-07-10T17:04:51.546920+0000 mgr.y (mgr.14144) 375 : cluster [DBG] pgmap v347: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:52.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:52 smithi116 bash[28351]: cluster 2023-07-10T17:04:51.546920+0000 mgr.y (mgr.14144) 375 : cluster [DBG] pgmap v347: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:52.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:52 smithi116 bash[23740]: cluster 2023-07-10T17:04:51.546920+0000 mgr.y (mgr.14144) 375 : cluster [DBG] pgmap v347: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:54.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:54 smithi192 bash[23730]: cluster 2023-07-10T17:04:53.548180+0000 mgr.y (mgr.14144) 376 : cluster [DBG] pgmap v348: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:54.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:54 smithi116 bash[23740]: cluster 2023-07-10T17:04:53.548180+0000 mgr.y (mgr.14144) 376 : cluster [DBG] pgmap v348: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:54.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:54 smithi116 bash[28351]: cluster 2023-07-10T17:04:53.548180+0000 mgr.y (mgr.14144) 376 : cluster [DBG] pgmap v348: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:56.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:56 smithi192 bash[23730]: cluster 2023-07-10T17:04:55.549153+0000 mgr.y (mgr.14144) 377 : cluster [DBG] pgmap v349: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:56.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:56 smithi116 bash[23740]: cluster 2023-07-10T17:04:55.549153+0000 mgr.y (mgr.14144) 377 : cluster [DBG] pgmap v349: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:56.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:56 smithi116 bash[28351]: cluster 2023-07-10T17:04:55.549153+0000 mgr.y (mgr.14144) 377 : cluster [DBG] pgmap v349: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:58.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:04:58 smithi192 bash[23730]: cluster 2023-07-10T17:04:57.550376+0000 mgr.y (mgr.14144) 378 : cluster [DBG] pgmap v350: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:58.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:04:58 smithi116 bash[23740]: cluster 2023-07-10T17:04:57.550376+0000 mgr.y (mgr.14144) 378 : cluster [DBG] pgmap v350: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:04:58.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:04:58 smithi116 bash[28351]: cluster 2023-07-10T17:04:57.550376+0000 mgr.y (mgr.14144) 378 : cluster [DBG] pgmap v350: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:00.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:00 smithi192 bash[23730]: cluster 2023-07-10T17:04:59.551680+0000 mgr.y (mgr.14144) 379 : cluster [DBG] pgmap v351: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:00.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:00 smithi116 bash[23740]: cluster 2023-07-10T17:04:59.551680+0000 mgr.y (mgr.14144) 379 : cluster [DBG] pgmap v351: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:00.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:00 smithi116 bash[28351]: cluster 2023-07-10T17:04:59.551680+0000 mgr.y (mgr.14144) 379 : cluster [DBG] pgmap v351: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:02.336 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2023-07-10T17:05:02.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:02 smithi192 bash[23730]: cluster 2023-07-10T17:05:01.552693+0000 mgr.y (mgr.14144) 380 : cluster [DBG] pgmap v352: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:02.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:02 smithi116 bash[23740]: cluster 2023-07-10T17:05:01.552693+0000 mgr.y (mgr.14144) 380 : cluster [DBG] pgmap v352: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:02.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:02 smithi116 bash[28351]: cluster 2023-07-10T17:05:01.552693+0000 mgr.y (mgr.14144) 380 : cluster [DBG] pgmap v352: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:04 smithi192 bash[23730]: cluster 2023-07-10T17:05:03.553754+0000 mgr.y (mgr.14144) 381 : cluster [DBG] pgmap v353: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:04 smithi192 bash[23730]: audit 2023-07-10T17:05:04.003700+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:04 smithi192 bash[23730]: audit 2023-07-10T17:05:04.243945+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:04.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:04 smithi116 bash[23740]: cluster 2023-07-10T17:05:03.553754+0000 mgr.y (mgr.14144) 381 : cluster [DBG] pgmap v353: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:04.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:04 smithi116 bash[23740]: audit 2023-07-10T17:05:04.003700+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:04.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:04 smithi116 bash[23740]: audit 2023-07-10T17:05:04.243945+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:04.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:04 smithi116 bash[28351]: cluster 2023-07-10T17:05:03.553754+0000 mgr.y (mgr.14144) 381 : cluster [DBG] pgmap v353: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:04.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:04 smithi116 bash[28351]: audit 2023-07-10T17:05:04.003700+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:04.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:04 smithi116 bash[28351]: audit 2023-07-10T17:05:04.243945+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:06.429 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2023-07-10T17:05:06.914 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:06 smithi116 bash[23740]: cluster 2023-07-10T17:05:05.554814+0000 mgr.y (mgr.14144) 382 : cluster [DBG] pgmap v354: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:06.914 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:06 smithi116 bash[23740]: audit 2023-07-10T17:05:05.713152+0000 mon.a (mon.0) 629 : audit [INF] from='client.? ' entity='client.admin' 2023-07-10T17:05:06.914 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:06 smithi116 bash[28351]: cluster 2023-07-10T17:05:05.554814+0000 mgr.y (mgr.14144) 382 : cluster [DBG] pgmap v354: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:06.915 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:06 smithi116 bash[28351]: audit 2023-07-10T17:05:05.713152+0000 mon.a (mon.0) 629 : audit [INF] from='client.? ' entity='client.admin' 2023-07-10T17:05:07.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:06 smithi192 bash[23730]: cluster 2023-07-10T17:05:05.554814+0000 mgr.y (mgr.14144) 382 : cluster [DBG] pgmap v354: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:07.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:06 smithi192 bash[23730]: audit 2023-07-10T17:05:05.713152+0000 mon.a (mon.0) 629 : audit [INF] from='client.? ' entity='client.admin' 2023-07-10T17:05:08.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:08 smithi116 bash[28351]: cluster 2023-07-10T17:05:07.556143+0000 mgr.y (mgr.14144) 383 : cluster [DBG] pgmap v355: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:08.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:08 smithi116 bash[23740]: cluster 2023-07-10T17:05:07.556143+0000 mgr.y (mgr.14144) 383 : cluster [DBG] pgmap v355: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:09.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:08 smithi192 bash[23730]: cluster 2023-07-10T17:05:07.556143+0000 mgr.y (mgr.14144) 383 : cluster [DBG] pgmap v355: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:10.566 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph config set global log_to_journald false --force' 2023-07-10T17:05:10.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:10 smithi116 bash[23740]: cluster 2023-07-10T17:05:09.557303+0000 mgr.y (mgr.14144) 384 : cluster [DBG] pgmap v356: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:10.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:10 smithi116 bash[23740]: audit 2023-07-10T17:05:09.894873+0000 mon.a (mon.0) 630 : audit [INF] from='client.? 172.21.15.116:0/2475829705' entity='client.admin' 2023-07-10T17:05:10.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:10 smithi116 bash[28351]: cluster 2023-07-10T17:05:09.557303+0000 mgr.y (mgr.14144) 384 : cluster [DBG] pgmap v356: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:10.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:10 smithi116 bash[28351]: audit 2023-07-10T17:05:09.894873+0000 mon.a (mon.0) 630 : audit [INF] from='client.? 172.21.15.116:0/2475829705' entity='client.admin' 2023-07-10T17:05:11.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:10 smithi192 bash[23730]: cluster 2023-07-10T17:05:09.557303+0000 mgr.y (mgr.14144) 384 : cluster [DBG] pgmap v356: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:11.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:10 smithi192 bash[23730]: audit 2023-07-10T17:05:09.894873+0000 mon.a (mon.0) 630 : audit [INF] from='client.? 172.21.15.116:0/2475829705' entity='client.admin' 2023-07-10T17:05:12.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:12 smithi116 bash[23740]: cluster 2023-07-10T17:05:11.558368+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v357: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:12.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:12 smithi116 bash[28351]: cluster 2023-07-10T17:05:11.558368+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v357: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:13.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:12 smithi192 bash[23730]: cluster 2023-07-10T17:05:11.558368+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v357: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:14.539 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2023-07-10T17:05:14.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:14 smithi192 bash[23730]: cluster 2023-07-10T17:05:13.559552+0000 mgr.y (mgr.14144) 386 : cluster [DBG] pgmap v358: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:14.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:14 smithi192 bash[23730]: audit 2023-07-10T17:05:13.857475+0000 mon.a (mon.0) 631 : audit [INF] from='client.? 172.21.15.116:0/1212715079' entity='client.admin' 2023-07-10T17:05:14.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:14 smithi192 bash[23730]: audit 2023-07-10T17:05:13.868114+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:14.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:14 smithi116 bash[28351]: cluster 2023-07-10T17:05:13.559552+0000 mgr.y (mgr.14144) 386 : cluster [DBG] pgmap v358: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:14.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:14 smithi116 bash[28351]: audit 2023-07-10T17:05:13.857475+0000 mon.a (mon.0) 631 : audit [INF] from='client.? 172.21.15.116:0/1212715079' entity='client.admin' 2023-07-10T17:05:14.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:14 smithi116 bash[28351]: audit 2023-07-10T17:05:13.868114+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:14.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:14 smithi116 bash[23740]: cluster 2023-07-10T17:05:13.559552+0000 mgr.y (mgr.14144) 386 : cluster [DBG] pgmap v358: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:14.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:14 smithi116 bash[23740]: audit 2023-07-10T17:05:13.857475+0000 mon.a (mon.0) 631 : audit [INF] from='client.? 172.21.15.116:0/1212715079' entity='client.admin' 2023-07-10T17:05:14.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:14 smithi116 bash[23740]: audit 2023-07-10T17:05:13.868114+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:16.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:16 smithi192 bash[23730]: cluster 2023-07-10T17:05:15.560554+0000 mgr.y (mgr.14144) 387 : cluster [DBG] pgmap v359: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:16.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:16 smithi116 bash[23740]: cluster 2023-07-10T17:05:15.560554+0000 mgr.y (mgr.14144) 387 : cluster [DBG] pgmap v359: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:16.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:16 smithi116 bash[28351]: cluster 2023-07-10T17:05:15.560554+0000 mgr.y (mgr.14144) 387 : cluster [DBG] pgmap v359: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:17.863 INFO:teuthology.orchestra.run.smithi116.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c 2023-07-10T17:05:18.693 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-10T17:05:18.704 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2023-07-10T17:05:18.704 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2023-07-10T17:05:18.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:18 smithi192 bash[23730]: cluster 2023-07-10T17:05:17.561428+0000 mgr.y (mgr.14144) 388 : cluster [DBG] pgmap v360: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:18.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:18 smithi192 bash[23730]: audit 2023-07-10T17:05:17.859039+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:18.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:18 smithi192 bash[23730]: audit 2023-07-10T17:05:17.868196+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:18.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:18 smithi116 bash[23740]: cluster 2023-07-10T17:05:17.561428+0000 mgr.y (mgr.14144) 388 : cluster [DBG] pgmap v360: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:18.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:18 smithi116 bash[23740]: audit 2023-07-10T17:05:17.859039+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:18.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:18 smithi116 bash[23740]: audit 2023-07-10T17:05:17.868196+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:18.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:18 smithi116 bash[28351]: cluster 2023-07-10T17:05:17.561428+0000 mgr.y (mgr.14144) 388 : cluster [DBG] pgmap v360: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:18.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:18 smithi116 bash[28351]: audit 2023-07-10T17:05:17.859039+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:18.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:18 smithi116 bash[28351]: audit 2023-07-10T17:05:17.868196+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:19.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:19 smithi116 bash[23740]: audit 2023-07-10T17:05:17.850885+0000 mgr.y (mgr.14144) 389 : audit [DBG] from='client.14916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:05:19.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:19 smithi116 bash[23740]: cephadm 2023-07-10T17:05:17.851742+0000 mgr.y (mgr.14144) 390 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c 2023-07-10T17:05:19.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:19 smithi116 bash[23740]: cephadm 2023-07-10T17:05:17.875732+0000 mgr.y (mgr.14144) 391 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c 2023-07-10T17:05:19.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:19 smithi116 bash[28351]: audit 2023-07-10T17:05:17.850885+0000 mgr.y (mgr.14144) 389 : audit [DBG] from='client.14916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:05:19.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:19 smithi116 bash[28351]: cephadm 2023-07-10T17:05:17.851742+0000 mgr.y (mgr.14144) 390 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c 2023-07-10T17:05:19.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:19 smithi116 bash[28351]: cephadm 2023-07-10T17:05:17.875732+0000 mgr.y (mgr.14144) 391 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c 2023-07-10T17:05:20.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:19 smithi192 bash[23730]: audit 2023-07-10T17:05:17.850885+0000 mgr.y (mgr.14144) 389 : audit [DBG] from='client.14916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:05:20.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:19 smithi192 bash[23730]: cephadm 2023-07-10T17:05:17.851742+0000 mgr.y (mgr.14144) 390 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c 2023-07-10T17:05:20.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:19 smithi192 bash[23730]: cephadm 2023-07-10T17:05:17.875732+0000 mgr.y (mgr.14144) 391 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c 2023-07-10T17:05:20.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:20 smithi116 bash[28351]: cluster 2023-07-10T17:05:19.562512+0000 mgr.y (mgr.14144) 392 : cluster [DBG] pgmap v361: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:20.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:20 smithi116 bash[28351]: audit 2023-07-10T17:05:19.635913+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:05:20.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:20 smithi116 bash[28351]: audit 2023-07-10T17:05:19.636104+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:05:20.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:20 smithi116 bash[23740]: cluster 2023-07-10T17:05:19.562512+0000 mgr.y (mgr.14144) 392 : cluster [DBG] pgmap v361: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:20.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:20 smithi116 bash[23740]: audit 2023-07-10T17:05:19.635913+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:05:20.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:20 smithi116 bash[23740]: audit 2023-07-10T17:05:19.636104+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:05:21.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:20 smithi192 bash[23730]: cluster 2023-07-10T17:05:19.562512+0000 mgr.y (mgr.14144) 392 : cluster [DBG] pgmap v361: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:21.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:20 smithi192 bash[23730]: audit 2023-07-10T17:05:19.635913+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:05:21.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:20 smithi192 bash[23730]: audit 2023-07-10T17:05:19.636104+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:05:23.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:22 smithi192 bash[23730]: cluster 2023-07-10T17:05:21.563344+0000 mgr.y (mgr.14144) 393 : cluster [DBG] pgmap v362: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:23.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:22 smithi116 bash[28351]: cluster 2023-07-10T17:05:21.563344+0000 mgr.y (mgr.14144) 393 : cluster [DBG] pgmap v362: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:23.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:22 smithi116 bash[23740]: cluster 2023-07-10T17:05:21.563344+0000 mgr.y (mgr.14144) 393 : cluster [DBG] pgmap v362: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:24.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:24 smithi192 bash[23730]: cluster 2023-07-10T17:05:23.564253+0000 mgr.y (mgr.14144) 394 : cluster [DBG] pgmap v363: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:24.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:24 smithi116 bash[28351]: cluster 2023-07-10T17:05:23.564253+0000 mgr.y (mgr.14144) 394 : cluster [DBG] pgmap v363: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:24.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:24 smithi116 bash[23740]: cluster 2023-07-10T17:05:23.564253+0000 mgr.y (mgr.14144) 394 : cluster [DBG] pgmap v363: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:26.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:26 smithi192 bash[23730]: cluster 2023-07-10T17:05:25.565005+0000 mgr.y (mgr.14144) 395 : cluster [DBG] pgmap v364: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:26.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:26 smithi116 bash[23740]: cluster 2023-07-10T17:05:25.565005+0000 mgr.y (mgr.14144) 395 : cluster [DBG] pgmap v364: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:26.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:26 smithi116 bash[28351]: cluster 2023-07-10T17:05:25.565005+0000 mgr.y (mgr.14144) 395 : cluster [DBG] pgmap v364: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:28.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:28 smithi192 bash[23730]: cluster 2023-07-10T17:05:27.565902+0000 mgr.y (mgr.14144) 396 : cluster [DBG] pgmap v365: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:28.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:28 smithi116 bash[23740]: cluster 2023-07-10T17:05:27.565902+0000 mgr.y (mgr.14144) 396 : cluster [DBG] pgmap v365: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:28.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:28 smithi116 bash[28351]: cluster 2023-07-10T17:05:27.565902+0000 mgr.y (mgr.14144) 396 : cluster [DBG] pgmap v365: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:30.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:30 smithi192 bash[23730]: cluster 2023-07-10T17:05:29.566732+0000 mgr.y (mgr.14144) 397 : cluster [DBG] pgmap v366: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:30.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:30 smithi116 bash[23740]: cluster 2023-07-10T17:05:29.566732+0000 mgr.y (mgr.14144) 397 : cluster [DBG] pgmap v366: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:30.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:30 smithi116 bash[28351]: cluster 2023-07-10T17:05:29.566732+0000 mgr.y (mgr.14144) 397 : cluster [DBG] pgmap v366: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:32.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:32 smithi116 bash[23740]: cluster 2023-07-10T17:05:31.567722+0000 mgr.y (mgr.14144) 398 : cluster [DBG] pgmap v367: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:32.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:32 smithi116 bash[28351]: cluster 2023-07-10T17:05:31.567722+0000 mgr.y (mgr.14144) 398 : cluster [DBG] pgmap v367: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:33.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:32 smithi192 bash[23730]: cluster 2023-07-10T17:05:31.567722+0000 mgr.y (mgr.14144) 398 : cluster [DBG] pgmap v367: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:34.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:34 smithi192 bash[23730]: cluster 2023-07-10T17:05:33.569023+0000 mgr.y (mgr.14144) 399 : cluster [DBG] pgmap v368: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:34.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:34 smithi116 bash[23740]: cluster 2023-07-10T17:05:33.569023+0000 mgr.y (mgr.14144) 399 : cluster [DBG] pgmap v368: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:34.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:34 smithi116 bash[28351]: cluster 2023-07-10T17:05:33.569023+0000 mgr.y (mgr.14144) 399 : cluster [DBG] pgmap v368: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:36.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:36 smithi192 bash[23730]: cluster 2023-07-10T17:05:35.570087+0000 mgr.y (mgr.14144) 400 : cluster [DBG] pgmap v369: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:36.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:36 smithi116 bash[23740]: cluster 2023-07-10T17:05:35.570087+0000 mgr.y (mgr.14144) 400 : cluster [DBG] pgmap v369: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:36.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:36 smithi116 bash[28351]: cluster 2023-07-10T17:05:35.570087+0000 mgr.y (mgr.14144) 400 : cluster [DBG] pgmap v369: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:38.189 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:05:38.875 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID 2023-07-10T17:05:38.875 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 running (4m) 2m ago 5m *:9093 *:9094 0.20.0 0881eb8f169f 54f60b4d9c19 2023-07-10T17:05:38.875 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 running (4m) 2m ago 4m *:3000 6.7.4 557c83e11646 ca7910fb45c5 2023-07-10T17:05:38.875 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 running (2m) 2m ago 2m - 24ecd6d5f14c 1f97a089df5d 2023-07-10T17:05:38.875 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 running (9m) 2m ago 9m *:8443 16.2.0 24ecd6d5f14c 98bd5a56dc04 2023-07-10T17:05:38.876 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 running (11m) 2m ago 11m *:9283 16.2.0 24ecd6d5f14c 9776722e922e 2023-07-10T17:05:38.876 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (11m) 2m ago 11m - 16.2.0 24ecd6d5f14c 33f26d450a67 2023-07-10T17:05:38.876 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (9m) 2m ago 9m - 16.2.0 24ecd6d5f14c 9ad4329e5488 2023-07-10T17:05:38.876 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (9m) 2m ago 9m - 16.2.0 24ecd6d5f14c 31e3ac12741b 2023-07-10T17:05:38.876 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 running (5m) 2m ago 5m *:9100 0.18.1 e5a616e4b9cf 8df3a5e8ba81 2023-07-10T17:05:38.876 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 running (5m) 2m ago 5m *:9100 0.18.1 e5a616e4b9cf db9b56cbd267 2023-07-10T17:05:38.876 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (8m) 2m ago 9m - 16.2.0 24ecd6d5f14c 4fa26367c830 2023-07-10T17:05:38.877 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (8m) 2m ago 8m - 16.2.0 24ecd6d5f14c c157509b3fd4 2023-07-10T17:05:38.877 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (8m) 2m ago 8m - 16.2.0 24ecd6d5f14c f1fd77b3c454 2023-07-10T17:05:38.877 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (7m) 2m ago 7m - 16.2.0 24ecd6d5f14c 60949d479c11 2023-07-10T17:05:38.877 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (7m) 2m ago 7m - 16.2.0 24ecd6d5f14c 75fc0b36aab8 2023-07-10T17:05:38.877 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (6m) 2m ago 6m - 16.2.0 24ecd6d5f14c d673065333dd 2023-07-10T17:05:38.877 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (6m) 2m ago 6m - 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:05:38.878 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (6m) 2m ago 6m - 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:05:38.878 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 running (4m) 2m ago 5m *:9095 2.18.1 de242295e225 95c5a7e6f5c5 2023-07-10T17:05:38.878 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 running (3m) 2m ago 3m *:8000 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:05:38.878 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 running (3m) 2m ago 3m *:8000 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:05:38.878 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 running (2m) 2m ago 2m *:80 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:05:38.878 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 running (2m) 2m ago 2m *:80 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:05:38.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:38 smithi116 bash[28351]: cluster 2023-07-10T17:05:37.571293+0000 mgr.y (mgr.14144) 401 : cluster [DBG] pgmap v370: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:38.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:38 smithi116 bash[23740]: cluster 2023-07-10T17:05:37.571293+0000 mgr.y (mgr.14144) 401 : cluster [DBG] pgmap v370: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:39.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:38 smithi192 bash[23730]: cluster 2023-07-10T17:05:37.571293+0000 mgr.y (mgr.14144) 401 : cluster [DBG] pgmap v370: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:39.240 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:05:39.241 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:05:39.241 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2023-07-10T17:05:39.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:05:39.241 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:05:39.241 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2 2023-07-10T17:05:39.242 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:05:39.242 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:05:39.242 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-10T17:05:39.242 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:05:39.242 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": {}, 2023-07-10T17:05:39.242 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:05:39.242 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:05:39.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:05:39.243 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:05:39.243 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 17 2023-07-10T17:05:39.243 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:05:39.243 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:05:39.591 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:05:39.591 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:05:39.592 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:05:39.592 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [], 2023-07-10T17:05:39.592 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "0/23 ceph daemons upgraded", 2023-07-10T17:05:39.592 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c image on host smithi192" 2023-07-10T17:05:39.592 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:05:39.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:39 smithi116 bash[23740]: audit 2023-07-10T17:05:38.172521+0000 mgr.y (mgr.14144) 402 : audit [DBG] from='client.14922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:05:39.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:39 smithi116 bash[23740]: audit 2023-07-10T17:05:38.525920+0000 mgr.y (mgr.14144) 403 : audit [DBG] from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:05:39.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:39 smithi116 bash[23740]: audit 2023-07-10T17:05:38.876616+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:39.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:39 smithi116 bash[23740]: audit 2023-07-10T17:05:38.880325+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:05:39.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:39 smithi116 bash[23740]: audit 2023-07-10T17:05:39.236879+0000 mon.a (mon.0) 639 : audit [DBG] from='client.? 172.21.15.116:0/2358293449' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:05:39.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:39 smithi116 bash[28351]: audit 2023-07-10T17:05:38.172521+0000 mgr.y (mgr.14144) 402 : audit [DBG] from='client.14922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:05:39.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:39 smithi116 bash[28351]: audit 2023-07-10T17:05:38.525920+0000 mgr.y (mgr.14144) 403 : audit [DBG] from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:05:39.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:39 smithi116 bash[28351]: audit 2023-07-10T17:05:38.876616+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:39.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:39 smithi116 bash[28351]: audit 2023-07-10T17:05:38.880325+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:05:39.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:39 smithi116 bash[28351]: audit 2023-07-10T17:05:39.236879+0000 mon.a (mon.0) 639 : audit [DBG] from='client.? 172.21.15.116:0/2358293449' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:05:39.972 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2023-07-10T17:05:40.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:39 smithi192 bash[23730]: audit 2023-07-10T17:05:38.172521+0000 mgr.y (mgr.14144) 402 : audit [DBG] from='client.14922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:05:40.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:39 smithi192 bash[23730]: audit 2023-07-10T17:05:38.525920+0000 mgr.y (mgr.14144) 403 : audit [DBG] from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:05:40.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:39 smithi192 bash[23730]: audit 2023-07-10T17:05:38.876616+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:05:40.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:39 smithi192 bash[23730]: audit 2023-07-10T17:05:38.880325+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:05:40.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:39 smithi192 bash[23730]: audit 2023-07-10T17:05:39.236879+0000 mon.a (mon.0) 639 : audit [DBG] from='client.? 172.21.15.116:0/2358293449' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:05:41.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:40 smithi192 bash[23730]: audit 2023-07-10T17:05:38.857143+0000 mgr.y (mgr.14144) 404 : audit [DBG] from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:05:41.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:40 smithi192 bash[23730]: cephadm 2023-07-10T17:05:38.877475+0000 mgr.y (mgr.14144) 405 : cephadm [INF] Upgrade: Target is version 18.1.2-30-g8507faa7 (unknown) 2023-07-10T17:05:41.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:40 smithi192 bash[23730]: cephadm 2023-07-10T17:05:38.877794+0000 mgr.y (mgr.14144) 406 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a5559d9b329bab12155f06d7e1994043cbd2e5807db218a236d708752ccf6200'] 2023-07-10T17:05:41.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:40 smithi192 bash[23730]: cephadm 2023-07-10T17:05:38.882372+0000 mgr.y (mgr.14144) 407 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-07-10T17:05:41.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:40 smithi192 bash[23730]: cephadm 2023-07-10T17:05:39.094127+0000 mgr.y (mgr.14144) 408 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c on smithi192 2023-07-10T17:05:41.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:40 smithi192 bash[23730]: cluster 2023-07-10T17:05:39.572531+0000 mgr.y (mgr.14144) 409 : cluster [DBG] pgmap v371: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:41.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:40 smithi192 bash[23730]: audit 2023-07-10T17:05:39.587116+0000 mgr.y (mgr.14144) 410 : audit [DBG] from='client.14946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:05:41.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:40 smithi192 bash[23730]: audit 2023-07-10T17:05:39.969289+0000 mon.a (mon.0) 640 : audit [DBG] from='client.? 172.21.15.116:0/3266392837' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:05:41.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:40 smithi116 bash[28351]: audit 2023-07-10T17:05:38.857143+0000 mgr.y (mgr.14144) 404 : audit [DBG] from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:05:41.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:40 smithi116 bash[28351]: cephadm 2023-07-10T17:05:38.877475+0000 mgr.y (mgr.14144) 405 : cephadm [INF] Upgrade: Target is version 18.1.2-30-g8507faa7 (unknown) 2023-07-10T17:05:41.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:40 smithi116 bash[28351]: cephadm 2023-07-10T17:05:38.877794+0000 mgr.y (mgr.14144) 406 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a5559d9b329bab12155f06d7e1994043cbd2e5807db218a236d708752ccf6200'] 2023-07-10T17:05:41.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:40 smithi116 bash[28351]: cephadm 2023-07-10T17:05:38.882372+0000 mgr.y (mgr.14144) 407 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-07-10T17:05:41.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:40 smithi116 bash[28351]: cephadm 2023-07-10T17:05:39.094127+0000 mgr.y (mgr.14144) 408 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c on smithi192 2023-07-10T17:05:41.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:40 smithi116 bash[28351]: cluster 2023-07-10T17:05:39.572531+0000 mgr.y (mgr.14144) 409 : cluster [DBG] pgmap v371: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:41.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:40 smithi116 bash[28351]: audit 2023-07-10T17:05:39.587116+0000 mgr.y (mgr.14144) 410 : audit [DBG] from='client.14946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:05:41.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:40 smithi116 bash[28351]: audit 2023-07-10T17:05:39.969289+0000 mon.a (mon.0) 640 : audit [DBG] from='client.? 172.21.15.116:0/3266392837' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:05:41.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:40 smithi116 bash[23740]: audit 2023-07-10T17:05:38.857143+0000 mgr.y (mgr.14144) 404 : audit [DBG] from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:05:41.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:40 smithi116 bash[23740]: cephadm 2023-07-10T17:05:38.877475+0000 mgr.y (mgr.14144) 405 : cephadm [INF] Upgrade: Target is version 18.1.2-30-g8507faa7 (unknown) 2023-07-10T17:05:41.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:40 smithi116 bash[23740]: cephadm 2023-07-10T17:05:38.877794+0000 mgr.y (mgr.14144) 406 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a5559d9b329bab12155f06d7e1994043cbd2e5807db218a236d708752ccf6200'] 2023-07-10T17:05:41.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:40 smithi116 bash[23740]: cephadm 2023-07-10T17:05:38.882372+0000 mgr.y (mgr.14144) 407 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-07-10T17:05:41.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:40 smithi116 bash[23740]: cephadm 2023-07-10T17:05:39.094127+0000 mgr.y (mgr.14144) 408 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c on smithi192 2023-07-10T17:05:41.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:40 smithi116 bash[23740]: cluster 2023-07-10T17:05:39.572531+0000 mgr.y (mgr.14144) 409 : cluster [DBG] pgmap v371: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:41.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:40 smithi116 bash[23740]: audit 2023-07-10T17:05:39.587116+0000 mgr.y (mgr.14144) 410 : audit [DBG] from='client.14946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:05:41.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:40 smithi116 bash[23740]: audit 2023-07-10T17:05:39.969289+0000 mon.a (mon.0) 640 : audit [DBG] from='client.? 172.21.15.116:0/3266392837' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:05:43.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:42 smithi192 bash[23730]: cluster 2023-07-10T17:05:41.573561+0000 mgr.y (mgr.14144) 411 : cluster [DBG] pgmap v372: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:43.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:42 smithi116 bash[23740]: cluster 2023-07-10T17:05:41.573561+0000 mgr.y (mgr.14144) 411 : cluster [DBG] pgmap v372: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:43.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:42 smithi116 bash[28351]: cluster 2023-07-10T17:05:41.573561+0000 mgr.y (mgr.14144) 411 : cluster [DBG] pgmap v372: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:44.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:44 smithi192 bash[23730]: cluster 2023-07-10T17:05:43.574837+0000 mgr.y (mgr.14144) 412 : cluster [DBG] pgmap v373: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:44.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:44 smithi116 bash[23740]: cluster 2023-07-10T17:05:43.574837+0000 mgr.y (mgr.14144) 412 : cluster [DBG] pgmap v373: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:44.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:44 smithi116 bash[28351]: cluster 2023-07-10T17:05:43.574837+0000 mgr.y (mgr.14144) 412 : cluster [DBG] pgmap v373: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:46.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:46 smithi116 bash[23740]: cluster 2023-07-10T17:05:45.575944+0000 mgr.y (mgr.14144) 413 : cluster [DBG] pgmap v374: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:46.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:46 smithi116 bash[28351]: cluster 2023-07-10T17:05:45.575944+0000 mgr.y (mgr.14144) 413 : cluster [DBG] pgmap v374: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:47.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:46 smithi192 bash[23730]: cluster 2023-07-10T17:05:45.575944+0000 mgr.y (mgr.14144) 413 : cluster [DBG] pgmap v374: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:48.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:48 smithi116 bash[23740]: cluster 2023-07-10T17:05:47.577139+0000 mgr.y (mgr.14144) 414 : cluster [DBG] pgmap v375: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:48.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:48 smithi116 bash[28351]: cluster 2023-07-10T17:05:47.577139+0000 mgr.y (mgr.14144) 414 : cluster [DBG] pgmap v375: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:49.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:48 smithi192 bash[23730]: cluster 2023-07-10T17:05:47.577139+0000 mgr.y (mgr.14144) 414 : cluster [DBG] pgmap v375: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:51.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:50 smithi192 bash[23730]: cluster 2023-07-10T17:05:49.578421+0000 mgr.y (mgr.14144) 415 : cluster [DBG] pgmap v376: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:51.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:50 smithi116 bash[23740]: cluster 2023-07-10T17:05:49.578421+0000 mgr.y (mgr.14144) 415 : cluster [DBG] pgmap v376: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:51.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:50 smithi116 bash[28351]: cluster 2023-07-10T17:05:49.578421+0000 mgr.y (mgr.14144) 415 : cluster [DBG] pgmap v376: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:53.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:52 smithi192 bash[23730]: cluster 2023-07-10T17:05:51.579457+0000 mgr.y (mgr.14144) 416 : cluster [DBG] pgmap v377: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:53.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:52 smithi116 bash[28351]: cluster 2023-07-10T17:05:51.579457+0000 mgr.y (mgr.14144) 416 : cluster [DBG] pgmap v377: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:53.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:52 smithi116 bash[23740]: cluster 2023-07-10T17:05:51.579457+0000 mgr.y (mgr.14144) 416 : cluster [DBG] pgmap v377: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:54.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:54 smithi192 bash[23730]: cluster 2023-07-10T17:05:53.580763+0000 mgr.y (mgr.14144) 417 : cluster [DBG] pgmap v378: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:54.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:54 smithi116 bash[28351]: cluster 2023-07-10T17:05:53.580763+0000 mgr.y (mgr.14144) 417 : cluster [DBG] pgmap v378: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:54.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:54 smithi116 bash[23740]: cluster 2023-07-10T17:05:53.580763+0000 mgr.y (mgr.14144) 417 : cluster [DBG] pgmap v378: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:56.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:56 smithi116 bash[23740]: cluster 2023-07-10T17:05:55.581893+0000 mgr.y (mgr.14144) 418 : cluster [DBG] pgmap v379: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:56.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:56 smithi116 bash[28351]: cluster 2023-07-10T17:05:55.581893+0000 mgr.y (mgr.14144) 418 : cluster [DBG] pgmap v379: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:57.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:56 smithi192 bash[23730]: cluster 2023-07-10T17:05:55.581893+0000 mgr.y (mgr.14144) 418 : cluster [DBG] pgmap v379: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:59.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:05:58 smithi192 bash[23730]: cluster 2023-07-10T17:05:57.583120+0000 mgr.y (mgr.14144) 419 : cluster [DBG] pgmap v380: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:59.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:05:58 smithi116 bash[23740]: cluster 2023-07-10T17:05:57.583120+0000 mgr.y (mgr.14144) 419 : cluster [DBG] pgmap v380: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:05:59.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:05:58 smithi116 bash[28351]: cluster 2023-07-10T17:05:57.583120+0000 mgr.y (mgr.14144) 419 : cluster [DBG] pgmap v380: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:01.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:00 smithi192 bash[23730]: cluster 2023-07-10T17:05:59.584507+0000 mgr.y (mgr.14144) 420 : cluster [DBG] pgmap v381: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:01.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:00 smithi116 bash[23740]: cluster 2023-07-10T17:05:59.584507+0000 mgr.y (mgr.14144) 420 : cluster [DBG] pgmap v381: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:01.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:00 smithi116 bash[28351]: cluster 2023-07-10T17:05:59.584507+0000 mgr.y (mgr.14144) 420 : cluster [DBG] pgmap v381: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:03.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:02 smithi192 bash[23730]: cluster 2023-07-10T17:06:01.585608+0000 mgr.y (mgr.14144) 421 : cluster [DBG] pgmap v382: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:03.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:02 smithi116 bash[23740]: cluster 2023-07-10T17:06:01.585608+0000 mgr.y (mgr.14144) 421 : cluster [DBG] pgmap v382: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:03.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:02 smithi116 bash[28351]: cluster 2023-07-10T17:06:01.585608+0000 mgr.y (mgr.14144) 421 : cluster [DBG] pgmap v382: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:04.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:04 smithi192 bash[23730]: cluster 2023-07-10T17:06:03.586946+0000 mgr.y (mgr.14144) 422 : cluster [DBG] pgmap v383: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:04.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:04 smithi116 bash[23740]: cluster 2023-07-10T17:06:03.586946+0000 mgr.y (mgr.14144) 422 : cluster [DBG] pgmap v383: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:04.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:04 smithi116 bash[28351]: cluster 2023-07-10T17:06:03.586946+0000 mgr.y (mgr.14144) 422 : cluster [DBG] pgmap v383: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:06.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:06 smithi192 bash[23730]: cluster 2023-07-10T17:06:05.588124+0000 mgr.y (mgr.14144) 423 : cluster [DBG] pgmap v384: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:06.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:06 smithi192 bash[23730]: audit 2023-07-10T17:06:06.284495+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:06:06.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:06 smithi192 bash[23730]: audit 2023-07-10T17:06:06.286297+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:06:06.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:06 smithi192 bash[23730]: audit 2023-07-10T17:06:06.287384+0000 mon.a (mon.0) 643 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T17:06:06.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:06 smithi192 bash[23730]: audit 2023-07-10T17:06:06.288317+0000 mon.a (mon.0) 644 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:06:07.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:06 smithi116 bash[23740]: cluster 2023-07-10T17:06:05.588124+0000 mgr.y (mgr.14144) 423 : cluster [DBG] pgmap v384: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:07.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:06 smithi116 bash[23740]: audit 2023-07-10T17:06:06.284495+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:06:07.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:06 smithi116 bash[23740]: audit 2023-07-10T17:06:06.286297+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:06:07.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:06 smithi116 bash[23740]: audit 2023-07-10T17:06:06.287384+0000 mon.a (mon.0) 643 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T17:06:07.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:06 smithi116 bash[23740]: audit 2023-07-10T17:06:06.288317+0000 mon.a (mon.0) 644 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:06:07.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:06 smithi116 bash[28351]: cluster 2023-07-10T17:06:05.588124+0000 mgr.y (mgr.14144) 423 : cluster [DBG] pgmap v384: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:07.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:06 smithi116 bash[28351]: audit 2023-07-10T17:06:06.284495+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:06:07.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:06 smithi116 bash[28351]: audit 2023-07-10T17:06:06.286297+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:06:07.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:06 smithi116 bash[28351]: audit 2023-07-10T17:06:06.287384+0000 mon.a (mon.0) 643 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T17:06:07.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:06 smithi116 bash[28351]: audit 2023-07-10T17:06:06.288317+0000 mon.a (mon.0) 644 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:06:08.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:07 smithi192 bash[23730]: cephadm 2023-07-10T17:06:06.274753+0000 mgr.y (mgr.14144) 424 : cephadm [INF] Upgrade: Updating mgr.x 2023-07-10T17:06:08.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:07 smithi192 bash[23730]: cephadm 2023-07-10T17:06:06.289134+0000 mgr.y (mgr.14144) 425 : cephadm [INF] Deploying daemon mgr.x on smithi192 2023-07-10T17:06:08.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:07 smithi116 bash[28351]: cephadm 2023-07-10T17:06:06.274753+0000 mgr.y (mgr.14144) 424 : cephadm [INF] Upgrade: Updating mgr.x 2023-07-10T17:06:08.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:07 smithi116 bash[28351]: cephadm 2023-07-10T17:06:06.289134+0000 mgr.y (mgr.14144) 425 : cephadm [INF] Deploying daemon mgr.x on smithi192 2023-07-10T17:06:08.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:07 smithi116 bash[23740]: cephadm 2023-07-10T17:06:06.274753+0000 mgr.y (mgr.14144) 424 : cephadm [INF] Upgrade: Updating mgr.x 2023-07-10T17:06:08.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:07 smithi116 bash[23740]: cephadm 2023-07-10T17:06:06.289134+0000 mgr.y (mgr.14144) 425 : cephadm [INF] Deploying daemon mgr.x on smithi192 2023-07-10T17:06:08.579 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:08 smithi192 systemd[1]: Stopping Ceph mgr.x for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:06:08.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:08 smithi192 bash[23730]: cluster 2023-07-10T17:06:07.589393+0000 mgr.y (mgr.14144) 426 : cluster [DBG] pgmap v385: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:09.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:08 smithi116 bash[23740]: cluster 2023-07-10T17:06:07.589393+0000 mgr.y (mgr.14144) 426 : cluster [DBG] pgmap v385: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:09.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:08 smithi116 bash[28351]: cluster 2023-07-10T17:06:07.589393+0000 mgr.y (mgr.14144) 426 : cluster [DBG] pgmap v385: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:09.429 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:09 smithi192 docker[42136]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-mgr.x 2023-07-10T17:06:09.430 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:09 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.x.service: Main process exited, code=exited, status=143/n/a 2023-07-10T17:06:09.430 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:09 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.x.service: Failed with result 'exit-code'. 2023-07-10T17:06:09.430 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:09 smithi192 systemd[1]: Stopped Ceph mgr.x for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:06:09.886 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:09 smithi192 systemd[1]: Started Ceph mgr.x for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:06:10.344 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:06:10.760 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:10 smithi192 bash[42233]: debug 2023-07-10T17:06:10.591+0000 7f7f5ba18700 1 -- 172.21.15.192:0/2361280079 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d3ac1c51e0 con 0x55d3acf44c00 2023-07-10T17:06:10.760 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:10 smithi192 bash[23730]: audit 2023-07-10T17:06:09.467674+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:06:10.761 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:10 smithi192 bash[23730]: cluster 2023-07-10T17:06:09.590749+0000 mgr.y (mgr.14144) 427 : cluster [DBG] pgmap v386: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:10.761 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:10 smithi192 bash[23730]: audit 2023-07-10T17:06:09.731889+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:06:10.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:10 smithi116 bash[23740]: audit 2023-07-10T17:06:09.467674+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:06:10.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:10 smithi116 bash[23740]: cluster 2023-07-10T17:06:09.590749+0000 mgr.y (mgr.14144) 427 : cluster [DBG] pgmap v386: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:10.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:10 smithi116 bash[23740]: audit 2023-07-10T17:06:09.731889+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:06:10.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:10 smithi116 bash[28351]: audit 2023-07-10T17:06:09.467674+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:06:10.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:10 smithi116 bash[28351]: cluster 2023-07-10T17:06:09.590749+0000 mgr.y (mgr.14144) 427 : cluster [DBG] pgmap v386: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:10.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:10 smithi116 bash[28351]: audit 2023-07-10T17:06:09.731889+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:06:11.041 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID 2023-07-10T17:06:11.042 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 running (4m) 3m ago 5m *:9093 *:9094 0.20.0 0881eb8f169f 54f60b4d9c19 2023-07-10T17:06:11.042 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 running (5m) 3m ago 5m *:3000 6.7.4 557c83e11646 ca7910fb45c5 2023-07-10T17:06:11.042 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 running (3m) 3m ago 3m - 24ecd6d5f14c 1f97a089df5d 2023-07-10T17:06:11.043 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 starting - - *:8443 2023-07-10T17:06:11.043 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 running (11m) 3m ago 11m *:9283 16.2.0 24ecd6d5f14c 9776722e922e 2023-07-10T17:06:11.043 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (11m) 3m ago 11m - 16.2.0 24ecd6d5f14c 33f26d450a67 2023-07-10T17:06:11.044 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (10m) 3m ago 10m - 16.2.0 24ecd6d5f14c 9ad4329e5488 2023-07-10T17:06:11.044 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (10m) 3m ago 10m - 16.2.0 24ecd6d5f14c 31e3ac12741b 2023-07-10T17:06:11.044 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 running (5m) 3m ago 5m *:9100 0.18.1 e5a616e4b9cf 8df3a5e8ba81 2023-07-10T17:06:11.044 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 running (5m) 3m ago 5m *:9100 0.18.1 e5a616e4b9cf db9b56cbd267 2023-07-10T17:06:11.045 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (9m) 3m ago 9m - 16.2.0 24ecd6d5f14c 4fa26367c830 2023-07-10T17:06:11.045 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (9m) 3m ago 9m - 16.2.0 24ecd6d5f14c c157509b3fd4 2023-07-10T17:06:11.045 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (8m) 3m ago 8m - 16.2.0 24ecd6d5f14c f1fd77b3c454 2023-07-10T17:06:11.045 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (8m) 3m ago 8m - 16.2.0 24ecd6d5f14c 60949d479c11 2023-07-10T17:06:11.046 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (7m) 3m ago 7m - 16.2.0 24ecd6d5f14c 75fc0b36aab8 2023-07-10T17:06:11.046 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (7m) 3m ago 7m - 16.2.0 24ecd6d5f14c d673065333dd 2023-07-10T17:06:11.046 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (7m) 3m ago 7m - 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:06:11.047 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (6m) 3m ago 6m - 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:06:11.047 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 running (4m) 3m ago 5m *:9095 2.18.1 de242295e225 95c5a7e6f5c5 2023-07-10T17:06:11.047 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 running (3m) 3m ago 3m *:8000 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:06:11.047 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 running (3m) 3m ago 3m *:8000 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:06:11.047 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 running (3m) 3m ago 3m *:80 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:06:11.047 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 running (3m) 3m ago 3m *:80 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:06:11.136 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:10 smithi192 bash[42233]: debug 2023-07-10T17:06:10.751+0000 7f7f65b66200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-10T17:06:11.411 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:06:11.412 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:06:11.412 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2023-07-10T17:06:11.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:06:11.412 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:06:11.412 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2 2023-07-10T17:06:11.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:06:11.413 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:06:11.413 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-10T17:06:11.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:06:11.413 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": {}, 2023-07-10T17:06:11.413 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:06:11.413 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:06:11.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:06:11.413 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:06:11.414 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 17 2023-07-10T17:06:11.414 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:06:11.414 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:06:11.739 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:11 smithi192 bash[42233]: debug 2023-07-10T17:06:11.631+0000 7f7f65b66200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-10T17:06:11.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:11 smithi192 bash[23730]: audit 2023-07-10T17:06:10.325815+0000 mgr.y (mgr.14144) 428 : audit [DBG] from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:11.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:11 smithi192 bash[23730]: audit 2023-07-10T17:06:10.671444+0000 mgr.y (mgr.14144) 429 : audit [DBG] from='client.24889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:11.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:11 smithi192 bash[23730]: audit 2023-07-10T17:06:11.407797+0000 mon.c (mon.1) 35 : audit [DBG] from='client.? 172.21.15.116:0/914534325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:06:11.769 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:06:11.769 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:06:11.770 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:06:11.770 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [], 2023-07-10T17:06:11.770 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "", 2023-07-10T17:06:11.770 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading mgr daemons" 2023-07-10T17:06:11.770 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:06:11.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:11 smithi116 bash[23740]: audit 2023-07-10T17:06:10.325815+0000 mgr.y (mgr.14144) 428 : audit [DBG] from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:11.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:11 smithi116 bash[23740]: audit 2023-07-10T17:06:10.671444+0000 mgr.y (mgr.14144) 429 : audit [DBG] from='client.24889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:11.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:11 smithi116 bash[23740]: audit 2023-07-10T17:06:11.407797+0000 mon.c (mon.1) 35 : audit [DBG] from='client.? 172.21.15.116:0/914534325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:06:11.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:11 smithi116 bash[28351]: audit 2023-07-10T17:06:10.325815+0000 mgr.y (mgr.14144) 428 : audit [DBG] from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:11.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:11 smithi116 bash[28351]: audit 2023-07-10T17:06:10.671444+0000 mgr.y (mgr.14144) 429 : audit [DBG] from='client.24889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:11.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:11 smithi116 bash[28351]: audit 2023-07-10T17:06:11.407797+0000 mon.c (mon.1) 35 : audit [DBG] from='client.? 172.21.15.116:0/914534325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:06:12.100 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:11 smithi192 bash[42233]: debug 2023-07-10T17:06:11.731+0000 7f7f65b66200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-10T17:06:12.101 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:11 smithi192 bash[42233]: debug 2023-07-10T17:06:11.831+0000 7f7f65b66200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-10T17:06:12.153 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2023-07-10T17:06:12.386 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:12 smithi192 bash[42233]: debug 2023-07-10T17:06:12.095+0000 7f7f65b66200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-10T17:06:12.848 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:12 smithi192 bash[42233]: debug 2023-07-10T17:06:12.439+0000 7f7f65b66200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-10T17:06:12.848 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:12 smithi192 bash[42233]: debug 2023-07-10T17:06:12.607+0000 7f7f65b66200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-10T17:06:12.849 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:12 smithi192 bash[23730]: audit 2023-07-10T17:06:11.017064+0000 mgr.y (mgr.14144) 430 : audit [DBG] from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:12.849 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:12 smithi192 bash[23730]: cluster 2023-07-10T17:06:11.591760+0000 mgr.y (mgr.14144) 431 : cluster [DBG] pgmap v387: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:12.849 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:12 smithi192 bash[23730]: audit 2023-07-10T17:06:11.765552+0000 mgr.y (mgr.14144) 432 : audit [DBG] from='client.24913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:12.849 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:12 smithi192 bash[23730]: audit 2023-07-10T17:06:12.150308+0000 mon.a (mon.0) 647 : audit [DBG] from='client.? 172.21.15.116:0/2948489645' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:06:12.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:12 smithi116 bash[23740]: audit 2023-07-10T17:06:11.017064+0000 mgr.y (mgr.14144) 430 : audit [DBG] from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:12.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:12 smithi116 bash[23740]: cluster 2023-07-10T17:06:11.591760+0000 mgr.y (mgr.14144) 431 : cluster [DBG] pgmap v387: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:12.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:12 smithi116 bash[23740]: audit 2023-07-10T17:06:11.765552+0000 mgr.y (mgr.14144) 432 : audit [DBG] from='client.24913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:12.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:12 smithi116 bash[23740]: audit 2023-07-10T17:06:12.150308+0000 mon.a (mon.0) 647 : audit [DBG] from='client.? 172.21.15.116:0/2948489645' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:06:12.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:12 smithi116 bash[28351]: audit 2023-07-10T17:06:11.017064+0000 mgr.y (mgr.14144) 430 : audit [DBG] from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:12.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:12 smithi116 bash[28351]: cluster 2023-07-10T17:06:11.591760+0000 mgr.y (mgr.14144) 431 : cluster [DBG] pgmap v387: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:12.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:12 smithi116 bash[28351]: audit 2023-07-10T17:06:11.765552+0000 mgr.y (mgr.14144) 432 : audit [DBG] from='client.24913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:12.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:12 smithi116 bash[28351]: audit 2023-07-10T17:06:12.150308+0000 mon.a (mon.0) 647 : audit [DBG] from='client.? 172.21.15.116:0/2948489645' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:06:13.136 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:12 smithi192 bash[42233]: debug 2023-07-10T17:06:12.839+0000 7f7f65b66200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-10T17:06:14.386 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:13 smithi192 bash[42233]: debug 2023-07-10T17:06:13.915+0000 7f7f65b66200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-10T17:06:14.386 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:14 smithi192 bash[42233]: debug 2023-07-10T17:06:14.031+0000 7f7f65b66200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-10T17:06:14.386 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:14 smithi192 bash[42233]: debug 2023-07-10T17:06:14.151+0000 7f7f65b66200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-10T17:06:14.862 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:14 smithi192 bash[42233]: debug 2023-07-10T17:06:14.623+0000 7f7f65b66200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-10T17:06:14.862 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:14 smithi192 bash[23730]: cluster 2023-07-10T17:06:13.592930+0000 mgr.y (mgr.14144) 433 : cluster [DBG] pgmap v388: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:14.862 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:14 smithi192 bash[23730]: audit 2023-07-10T17:06:13.870501+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:06:14.863 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:14 smithi192 bash[23730]: audit 2023-07-10T17:06:14.107802+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:06:14.863 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:14 smithi192 bash[23730]: audit 2023-07-10T17:06:14.117284+0000 mon.a (mon.0) 650 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:06:14.863 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:14 smithi192 bash[23730]: audit 2023-07-10T17:06:14.122438+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2023-07-10T17:06:14.863 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:14 smithi192 bash[23730]: cluster 2023-07-10T17:06:14.130714+0000 mon.a (mon.0) 652 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2023-07-10T17:06:14.863 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:14 smithi192 bash[23730]: cluster 2023-07-10T17:06:14.157373+0000 mon.a (mon.0) 653 : cluster [DBG] Standby manager daemon y started 2023-07-10T17:06:14.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:14 smithi116 bash[23740]: cluster 2023-07-10T17:06:13.592930+0000 mgr.y (mgr.14144) 433 : cluster [DBG] pgmap v388: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:14.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:14 smithi116 bash[23740]: audit 2023-07-10T17:06:13.870501+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:06:14.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:14 smithi116 bash[23740]: audit 2023-07-10T17:06:14.107802+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:06:14.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:14 smithi116 bash[23740]: audit 2023-07-10T17:06:14.117284+0000 mon.a (mon.0) 650 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:06:14.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:14 smithi116 bash[23740]: audit 2023-07-10T17:06:14.122438+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2023-07-10T17:06:14.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:14 smithi116 bash[23740]: cluster 2023-07-10T17:06:14.130714+0000 mon.a (mon.0) 652 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2023-07-10T17:06:14.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:14 smithi116 bash[23740]: cluster 2023-07-10T17:06:14.157373+0000 mon.a (mon.0) 653 : cluster [DBG] Standby manager daemon y started 2023-07-10T17:06:14.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:14 smithi116 bash[28351]: cluster 2023-07-10T17:06:13.592930+0000 mgr.y (mgr.14144) 433 : cluster [DBG] pgmap v388: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:14.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:14 smithi116 bash[28351]: audit 2023-07-10T17:06:13.870501+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:06:14.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:14 smithi116 bash[28351]: audit 2023-07-10T17:06:14.107802+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' 2023-07-10T17:06:14.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:14 smithi116 bash[28351]: audit 2023-07-10T17:06:14.117284+0000 mon.a (mon.0) 650 : audit [DBG] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:06:14.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:14 smithi116 bash[28351]: audit 2023-07-10T17:06:14.122438+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2023-07-10T17:06:14.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:14 smithi116 bash[28351]: cluster 2023-07-10T17:06:14.130714+0000 mon.a (mon.0) 652 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2023-07-10T17:06:14.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:14 smithi116 bash[28351]: cluster 2023-07-10T17:06:14.157373+0000 mon.a (mon.0) 653 : cluster [DBG] Standby manager daemon y started 2023-07-10T17:06:15.136 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:14 smithi192 bash[42233]: debug 2023-07-10T17:06:14.855+0000 7f7f65b66200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-10T17:06:15.553 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:06:15 smithi116 bash[24013]: ignoring --setuser ceph since I am not root 2023-07-10T17:06:15.553 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:06:15 smithi116 bash[24013]: ignoring --setgroup ceph since I am not root 2023-07-10T17:06:15.553 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:06:15 smithi116 bash[24013]: debug 2023-07-10T17:06:15.157+0000 7fef323e6700 1 -- 172.21.15.116:0/2777937231 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5561dcce84e0 con 0x5561dcdd8c00 2023-07-10T17:06:15.554 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:15 smithi192 bash[42233]: debug 2023-07-10T17:06:15.263+0000 7f7f65b66200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-10T17:06:15.555 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:15 smithi116 bash[23740]: cephadm 2023-07-10T17:06:14.119073+0000 mgr.y (mgr.14144) 434 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-07-10T17:06:15.555 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:15 smithi116 bash[23740]: cephadm 2023-07-10T17:06:14.121934+0000 mgr.y (mgr.14144) 435 : cephadm [INF] Failing over to other MGR 2023-07-10T17:06:15.555 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:15 smithi116 bash[23740]: audit 2023-07-10T17:06:15.120501+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2023-07-10T17:06:15.555 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:15 smithi116 bash[23740]: cluster 2023-07-10T17:06:15.120668+0000 mon.a (mon.0) 655 : cluster [DBG] mgrmap e16: x(active, starting, since 0.996992s), standbys: y 2023-07-10T17:06:15.555 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:15 smithi116 bash[28351]: cephadm 2023-07-10T17:06:14.119073+0000 mgr.y (mgr.14144) 434 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-07-10T17:06:15.556 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:15 smithi116 bash[28351]: cephadm 2023-07-10T17:06:14.121934+0000 mgr.y (mgr.14144) 435 : cephadm [INF] Failing over to other MGR 2023-07-10T17:06:15.556 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:15 smithi116 bash[28351]: audit 2023-07-10T17:06:15.120501+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2023-07-10T17:06:15.556 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:15 smithi116 bash[28351]: cluster 2023-07-10T17:06:15.120668+0000 mon.a (mon.0) 655 : cluster [DBG] mgrmap e16: x(active, starting, since 0.996992s), standbys: y 2023-07-10T17:06:15.886 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:15 smithi192 bash[42233]: debug 2023-07-10T17:06:15.779+0000 7f7f65b66200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-10T17:06:15.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:15 smithi192 bash[23730]: cephadm 2023-07-10T17:06:14.119073+0000 mgr.y (mgr.14144) 434 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-07-10T17:06:15.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:15 smithi192 bash[23730]: cephadm 2023-07-10T17:06:14.121934+0000 mgr.y (mgr.14144) 435 : cephadm [INF] Failing over to other MGR 2023-07-10T17:06:15.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:15 smithi192 bash[23730]: audit 2023-07-10T17:06:15.120501+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14144 172.21.15.116:0/307139665' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2023-07-10T17:06:15.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:15 smithi192 bash[23730]: cluster 2023-07-10T17:06:15.120668+0000 mon.a (mon.0) 655 : cluster [DBG] mgrmap e16: x(active, starting, since 0.996992s), standbys: y 2023-07-10T17:06:16.636 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:16 smithi192 bash[42233]: debug 2023-07-10T17:06:16.327+0000 7f7f65b66200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-10T17:06:16.637 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:16 smithi192 bash[42233]: debug 2023-07-10T17:06:16.435+0000 7f7f65b66200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-10T17:06:17.042 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:16 smithi192 bash[42233]: debug 2023-07-10T17:06:16.783+0000 7f7f65b66200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-10T17:06:17.042 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:16 smithi192 bash[42233]: debug 2023-07-10T17:06:16.895+0000 7f7f65b66200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-10T17:06:17.386 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:17 smithi192 bash[42233]: debug 2023-07-10T17:06:17.035+0000 7f7f65b66200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-10T17:06:17.387 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:17 smithi192 bash[42233]: debug 2023-07-10T17:06:17.155+0000 7f7f65b66200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-10T17:06:17.387 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:17 smithi192 bash[42233]: debug 2023-07-10T17:06:17.275+0000 7f7f65b66200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-10T17:06:17.886 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:17 smithi192 bash[42233]: debug 2023-07-10T17:06:17.463+0000 7f7f65b66200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-10T17:06:17.886 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:17 smithi192 bash[42233]: debug 2023-07-10T17:06:17.587+0000 7f7f65b66200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-10T17:06:19.136 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:06:18 smithi192 bash[42233]: debug 2023-07-10T17:06:18.791+0000 7f7f65b66200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-10T17:06:19.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:19 smithi192 bash[23730]: cluster 2023-07-10T17:06:18.816687+0000 mon.a (mon.0) 656 : cluster [INF] Active manager daemon x restarted 2023-07-10T17:06:19.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:19 smithi192 bash[23730]: cluster 2023-07-10T17:06:18.817949+0000 mon.a (mon.0) 657 : cluster [INF] Activating manager daemon x 2023-07-10T17:06:19.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:19 smithi192 bash[23730]: audit 2023-07-10T17:06:18.819858+0000 mon.b (mon.2) 22 : audit [DBG] from='mgr.? 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-10T17:06:19.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:19 smithi192 bash[23730]: cluster 2023-07-10T17:06:18.825799+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2023-07-10T17:06:19.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:19 smithi192 bash[23730]: audit 2023-07-10T17:06:18.827965+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:06:19.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:19 smithi192 bash[23730]: audit 2023-07-10T17:06:18.830044+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.? 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-10T17:06:19.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:19 smithi192 bash[23730]: audit 2023-07-10T17:06:18.830429+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.? 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:06:19.829 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:19 smithi116 bash[23740]: cluster 2023-07-10T17:06:18.816687+0000 mon.a (mon.0) 656 : cluster [INF] Active manager daemon x restarted 2023-07-10T17:06:19.829 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:19 smithi116 bash[23740]: cluster 2023-07-10T17:06:18.817949+0000 mon.a (mon.0) 657 : cluster [INF] Activating manager daemon x 2023-07-10T17:06:19.829 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:19 smithi116 bash[23740]: audit 2023-07-10T17:06:18.819858+0000 mon.b (mon.2) 22 : audit [DBG] from='mgr.? 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-10T17:06:19.829 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:19 smithi116 bash[23740]: cluster 2023-07-10T17:06:18.825799+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2023-07-10T17:06:19.829 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:19 smithi116 bash[23740]: audit 2023-07-10T17:06:18.827965+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:06:19.829 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:19 smithi116 bash[23740]: audit 2023-07-10T17:06:18.830044+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.? 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-10T17:06:19.830 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:19 smithi116 bash[23740]: audit 2023-07-10T17:06:18.830429+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.? 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:06:19.830 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:19 smithi116 bash[28351]: cluster 2023-07-10T17:06:18.816687+0000 mon.a (mon.0) 656 : cluster [INF] Active manager daemon x restarted 2023-07-10T17:06:19.830 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:19 smithi116 bash[28351]: cluster 2023-07-10T17:06:18.817949+0000 mon.a (mon.0) 657 : cluster [INF] Activating manager daemon x 2023-07-10T17:06:19.830 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:19 smithi116 bash[28351]: audit 2023-07-10T17:06:18.819858+0000 mon.b (mon.2) 22 : audit [DBG] from='mgr.? 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-10T17:06:19.830 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:19 smithi116 bash[28351]: cluster 2023-07-10T17:06:18.825799+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2023-07-10T17:06:19.830 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:19 smithi116 bash[28351]: audit 2023-07-10T17:06:18.827965+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:06:19.830 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:19 smithi116 bash[28351]: audit 2023-07-10T17:06:18.830044+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.? 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-10T17:06:19.831 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:19 smithi116 bash[28351]: audit 2023-07-10T17:06:18.830429+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.? 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:06:20.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: cluster 2023-07-10T17:06:19.558237+0000 mon.a (mon.0) 659 : cluster [DBG] mgrmap e17: x(active, starting, since 0.740579s), standbys: y 2023-07-10T17:06:20.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.569528+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:06:20.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.569983+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:06:20.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.570267+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:06:20.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.571440+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-10T17:06:20.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.571799+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-10T17:06:20.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.572093+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T17:06:20.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.572452+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T17:06:20.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.572744+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T17:06:20.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.573024+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T17:06:20.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.573275+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T17:06:20.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.573540+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T17:06:20.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.573808+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T17:06:20.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.574060+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T17:06:20.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.574999+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-10T17:06:20.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.575333+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-10T17:06:20.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.576142+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-10T17:06:20.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: cluster 2023-07-10T17:06:19.603072+0000 mon.a (mon.0) 660 : cluster [INF] Manager daemon x is now available 2023-07-10T17:06:20.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.636651+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.658236+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.666255+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.677104+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:06:20.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.698623+0000 mon.b (mon.2) 43 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-10T17:06:20.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.698642+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-10T17:06:20.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.730223+0000 mon.b (mon.2) 44 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:06:20.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.730392+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:06:20.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.758335+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.765052+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-10T17:06:20.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:19.765345+0000 mon.b (mon.2) 45 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-10T17:06:20.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:20.063573+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:20 smithi192 bash[23730]: audit 2023-07-10T17:06:20.072285+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: cluster 2023-07-10T17:06:19.558237+0000 mon.a (mon.0) 659 : cluster [DBG] mgrmap e17: x(active, starting, since 0.740579s), standbys: y 2023-07-10T17:06:20.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.569528+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:06:20.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.569983+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:06:20.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.570267+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:06:20.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.571440+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-10T17:06:20.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.571799+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-10T17:06:20.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.572093+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T17:06:20.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.572452+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T17:06:20.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.572744+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T17:06:20.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.573024+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T17:06:20.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.573275+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T17:06:20.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.573540+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T17:06:20.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.573808+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T17:06:20.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.574060+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T17:06:20.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.574999+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-10T17:06:20.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.575333+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-10T17:06:20.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.576142+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-10T17:06:20.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: cluster 2023-07-10T17:06:19.603072+0000 mon.a (mon.0) 660 : cluster [INF] Manager daemon x is now available 2023-07-10T17:06:20.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.636651+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: cluster 2023-07-10T17:06:19.558237+0000 mon.a (mon.0) 659 : cluster [DBG] mgrmap e17: x(active, starting, since 0.740579s), standbys: y 2023-07-10T17:06:20.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.569528+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:06:20.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.569983+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:06:20.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.570267+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:06:20.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.571440+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-10T17:06:20.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.571799+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-10T17:06:20.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.572093+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T17:06:20.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.572452+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T17:06:20.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.572744+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T17:06:20.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.573024+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T17:06:20.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.573275+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T17:06:20.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.573540+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T17:06:20.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.573808+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T17:06:20.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.574060+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T17:06:20.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.574999+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-10T17:06:20.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.575333+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-10T17:06:20.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.576142+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-10T17:06:20.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: cluster 2023-07-10T17:06:19.603072+0000 mon.a (mon.0) 660 : cluster [INF] Manager daemon x is now available 2023-07-10T17:06:20.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.636651+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.658236+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.666255+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.677104+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:06:20.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.698623+0000 mon.b (mon.2) 43 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-10T17:06:20.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.698642+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-10T17:06:20.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.730223+0000 mon.b (mon.2) 44 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:06:20.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.730392+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:06:20.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.758335+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.765052+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-10T17:06:20.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:19.765345+0000 mon.b (mon.2) 45 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-10T17:06:20.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:20.063573+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[28351]: audit 2023-07-10T17:06:20.072285+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.658236+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.666255+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.677104+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:06:20.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.698623+0000 mon.b (mon.2) 43 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-10T17:06:20.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.698642+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-10T17:06:20.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.730223+0000 mon.b (mon.2) 44 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:06:20.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.730392+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:06:20.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.758335+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.765052+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-10T17:06:20.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:19.765345+0000 mon.b (mon.2) 45 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-10T17:06:20.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:20.063573+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:20.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:20 smithi116 bash[23740]: audit 2023-07-10T17:06:20.072285+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:21.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:21 smithi192 bash[23730]: cephadm 2023-07-10T17:06:19.629904+0000 mgr.x (mgr.24857) 1 : cephadm [INF] inventory: adjusted host smithi116 addr 'smithi116' -> '172.21.15.116' 2023-07-10T17:06:21.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:21 smithi192 bash[23730]: cephadm 2023-07-10T17:06:19.659613+0000 mgr.x (mgr.24857) 2 : cephadm [INF] Queued rgw.foo for migration 2023-07-10T17:06:21.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:21 smithi192 bash[23730]: cephadm 2023-07-10T17:06:19.667698+0000 mgr.x (mgr.24857) 3 : cephadm [INF] Queued rgw.smpl for migration 2023-07-10T17:06:21.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:21 smithi192 bash[23730]: cephadm 2023-07-10T17:06:20.138482+0000 mgr.x (mgr.24857) 4 : cephadm [INF] Deploying cephadm binary to smithi192 2023-07-10T17:06:21.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:21 smithi192 bash[23730]: cephadm 2023-07-10T17:06:20.148426+0000 mgr.x (mgr.24857) 5 : cephadm [INF] Deploying cephadm binary to smithi116 2023-07-10T17:06:21.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:21 smithi192 bash[23730]: cluster 2023-07-10T17:06:20.581399+0000 mon.a (mon.0) 670 : cluster [DBG] mgrmap e18: x(active, since 1.76374s), standbys: y 2023-07-10T17:06:21.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:21 smithi192 bash[23730]: audit 2023-07-10T17:06:20.671084+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2023-07-10T17:06:21.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:21 smithi192 bash[23730]: cluster 2023-07-10T17:06:20.671460+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2023-07-10T17:06:21.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:21 smithi192 bash[23730]: audit 2023-07-10T17:06:20.682289+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-10T17:06:21.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:21 smithi192 bash[23730]: audit 2023-07-10T17:06:20.682407+0000 mon.b (mon.2) 46 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-10T17:06:21.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:21 smithi192 bash[23730]: cluster 2023-07-10T17:06:20.989533+0000 mon.a (mon.0) 674 : cluster [DBG] Standby manager daemon y restarted 2023-07-10T17:06:21.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:21 smithi192 bash[23730]: cluster 2023-07-10T17:06:20.989758+0000 mon.a (mon.0) 675 : cluster [DBG] Standby manager daemon y started 2023-07-10T17:06:21.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:21 smithi192 bash[23730]: audit 2023-07-10T17:06:20.991359+0000 mon.c (mon.1) 36 : audit [DBG] from='mgr.? 172.21.15.116:0/382458661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-10T17:06:21.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:21 smithi192 bash[23730]: audit 2023-07-10T17:06:20.997310+0000 mon.c (mon.1) 37 : audit [DBG] from='mgr.? 172.21.15.116:0/382458661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:06:21.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:21 smithi192 bash[23730]: audit 2023-07-10T17:06:20.998791+0000 mon.c (mon.1) 38 : audit [DBG] from='mgr.? 172.21.15.116:0/382458661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-10T17:06:21.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:21 smithi192 bash[23730]: audit 2023-07-10T17:06:20.999478+0000 mon.c (mon.1) 39 : audit [DBG] from='mgr.? 172.21.15.116:0/382458661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:06:21.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[23740]: cephadm 2023-07-10T17:06:19.629904+0000 mgr.x (mgr.24857) 1 : cephadm [INF] inventory: adjusted host smithi116 addr 'smithi116' -> '172.21.15.116' 2023-07-10T17:06:21.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[23740]: cephadm 2023-07-10T17:06:19.659613+0000 mgr.x (mgr.24857) 2 : cephadm [INF] Queued rgw.foo for migration 2023-07-10T17:06:21.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[23740]: cephadm 2023-07-10T17:06:19.667698+0000 mgr.x (mgr.24857) 3 : cephadm [INF] Queued rgw.smpl for migration 2023-07-10T17:06:21.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[23740]: cephadm 2023-07-10T17:06:20.138482+0000 mgr.x (mgr.24857) 4 : cephadm [INF] Deploying cephadm binary to smithi192 2023-07-10T17:06:21.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[23740]: cephadm 2023-07-10T17:06:20.148426+0000 mgr.x (mgr.24857) 5 : cephadm [INF] Deploying cephadm binary to smithi116 2023-07-10T17:06:21.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[23740]: cluster 2023-07-10T17:06:20.581399+0000 mon.a (mon.0) 670 : cluster [DBG] mgrmap e18: x(active, since 1.76374s), standbys: y 2023-07-10T17:06:21.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[23740]: audit 2023-07-10T17:06:20.671084+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2023-07-10T17:06:21.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[23740]: cluster 2023-07-10T17:06:20.671460+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2023-07-10T17:06:21.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[23740]: audit 2023-07-10T17:06:20.682289+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-10T17:06:21.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[23740]: audit 2023-07-10T17:06:20.682407+0000 mon.b (mon.2) 46 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-10T17:06:21.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[23740]: cluster 2023-07-10T17:06:20.989533+0000 mon.a (mon.0) 674 : cluster [DBG] Standby manager daemon y restarted 2023-07-10T17:06:21.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[23740]: cluster 2023-07-10T17:06:20.989758+0000 mon.a (mon.0) 675 : cluster [DBG] Standby manager daemon y started 2023-07-10T17:06:21.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[23740]: audit 2023-07-10T17:06:20.991359+0000 mon.c (mon.1) 36 : audit [DBG] from='mgr.? 172.21.15.116:0/382458661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-10T17:06:21.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[23740]: audit 2023-07-10T17:06:20.997310+0000 mon.c (mon.1) 37 : audit [DBG] from='mgr.? 172.21.15.116:0/382458661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:06:21.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[23740]: audit 2023-07-10T17:06:20.998791+0000 mon.c (mon.1) 38 : audit [DBG] from='mgr.? 172.21.15.116:0/382458661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-10T17:06:21.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[23740]: audit 2023-07-10T17:06:20.999478+0000 mon.c (mon.1) 39 : audit [DBG] from='mgr.? 172.21.15.116:0/382458661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:06:21.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[28351]: cephadm 2023-07-10T17:06:19.629904+0000 mgr.x (mgr.24857) 1 : cephadm [INF] inventory: adjusted host smithi116 addr 'smithi116' -> '172.21.15.116' 2023-07-10T17:06:21.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[28351]: cephadm 2023-07-10T17:06:19.659613+0000 mgr.x (mgr.24857) 2 : cephadm [INF] Queued rgw.foo for migration 2023-07-10T17:06:21.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[28351]: cephadm 2023-07-10T17:06:19.667698+0000 mgr.x (mgr.24857) 3 : cephadm [INF] Queued rgw.smpl for migration 2023-07-10T17:06:21.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[28351]: cephadm 2023-07-10T17:06:20.138482+0000 mgr.x (mgr.24857) 4 : cephadm [INF] Deploying cephadm binary to smithi192 2023-07-10T17:06:21.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[28351]: cephadm 2023-07-10T17:06:20.148426+0000 mgr.x (mgr.24857) 5 : cephadm [INF] Deploying cephadm binary to smithi116 2023-07-10T17:06:21.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[28351]: cluster 2023-07-10T17:06:20.581399+0000 mon.a (mon.0) 670 : cluster [DBG] mgrmap e18: x(active, since 1.76374s), standbys: y 2023-07-10T17:06:21.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[28351]: audit 2023-07-10T17:06:20.671084+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2023-07-10T17:06:21.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[28351]: cluster 2023-07-10T17:06:20.671460+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2023-07-10T17:06:21.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[28351]: audit 2023-07-10T17:06:20.682289+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-10T17:06:21.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[28351]: audit 2023-07-10T17:06:20.682407+0000 mon.b (mon.2) 46 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-10T17:06:21.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[28351]: cluster 2023-07-10T17:06:20.989533+0000 mon.a (mon.0) 674 : cluster [DBG] Standby manager daemon y restarted 2023-07-10T17:06:21.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[28351]: cluster 2023-07-10T17:06:20.989758+0000 mon.a (mon.0) 675 : cluster [DBG] Standby manager daemon y started 2023-07-10T17:06:21.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[28351]: audit 2023-07-10T17:06:20.991359+0000 mon.c (mon.1) 36 : audit [DBG] from='mgr.? 172.21.15.116:0/382458661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-10T17:06:21.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[28351]: audit 2023-07-10T17:06:20.997310+0000 mon.c (mon.1) 37 : audit [DBG] from='mgr.? 172.21.15.116:0/382458661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:06:21.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[28351]: audit 2023-07-10T17:06:20.998791+0000 mon.c (mon.1) 38 : audit [DBG] from='mgr.? 172.21.15.116:0/382458661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-10T17:06:21.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:21 smithi116 bash[28351]: audit 2023-07-10T17:06:20.999478+0000 mon.c (mon.1) 39 : audit [DBG] from='mgr.? 172.21.15.116:0/382458661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:06:22.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:22 smithi192 bash[23730]: cluster 2023-07-10T17:06:20.615971+0000 mgr.x (mgr.24857) 6 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:22.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:22 smithi192 bash[23730]: cephadm 2023-07-10T17:06:20.919360+0000 mgr.x (mgr.24857) 7 : cephadm [INF] [10/Jul/2023:17:06:20] ENGINE Bus STARTING 2023-07-10T17:06:22.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:22 smithi192 bash[23730]: cephadm 2023-07-10T17:06:21.070308+0000 mgr.x (mgr.24857) 8 : cephadm [INF] [10/Jul/2023:17:06:21] ENGINE Serving on http://172.21.15.192:8765 2023-07-10T17:06:22.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:22 smithi192 bash[23730]: cephadm 2023-07-10T17:06:21.293781+0000 mgr.x (mgr.24857) 9 : cephadm [INF] [10/Jul/2023:17:06:21] ENGINE Serving on https://172.21.15.192:7150 2023-07-10T17:06:22.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:22 smithi192 bash[23730]: cephadm 2023-07-10T17:06:21.294294+0000 mgr.x (mgr.24857) 10 : cephadm [INF] [10/Jul/2023:17:06:21] ENGINE Bus STARTED 2023-07-10T17:06:22.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:22 smithi192 bash[23730]: cluster 2023-07-10T17:06:21.575514+0000 mgr.x (mgr.24857) 11 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:22.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:22 smithi192 bash[23730]: audit 2023-07-10T17:06:21.680311+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-10T17:06:22.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:22 smithi192 bash[23730]: cluster 2023-07-10T17:06:21.680572+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2023-07-10T17:06:22.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:22 smithi192 bash[23730]: cluster 2023-07-10T17:06:21.688603+0000 mon.a (mon.0) 678 : cluster [DBG] mgrmap e19: x(active, since 2s), standbys: y 2023-07-10T17:06:22.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[23740]: cluster 2023-07-10T17:06:20.615971+0000 mgr.x (mgr.24857) 6 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:22.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[23740]: cephadm 2023-07-10T17:06:20.919360+0000 mgr.x (mgr.24857) 7 : cephadm [INF] [10/Jul/2023:17:06:20] ENGINE Bus STARTING 2023-07-10T17:06:22.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[23740]: cephadm 2023-07-10T17:06:21.070308+0000 mgr.x (mgr.24857) 8 : cephadm [INF] [10/Jul/2023:17:06:21] ENGINE Serving on http://172.21.15.192:8765 2023-07-10T17:06:22.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[23740]: cephadm 2023-07-10T17:06:21.293781+0000 mgr.x (mgr.24857) 9 : cephadm [INF] [10/Jul/2023:17:06:21] ENGINE Serving on https://172.21.15.192:7150 2023-07-10T17:06:22.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[23740]: cephadm 2023-07-10T17:06:21.294294+0000 mgr.x (mgr.24857) 10 : cephadm [INF] [10/Jul/2023:17:06:21] ENGINE Bus STARTED 2023-07-10T17:06:22.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[23740]: cluster 2023-07-10T17:06:21.575514+0000 mgr.x (mgr.24857) 11 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:22.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[23740]: audit 2023-07-10T17:06:21.680311+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-10T17:06:22.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[23740]: cluster 2023-07-10T17:06:21.680572+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2023-07-10T17:06:22.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[23740]: cluster 2023-07-10T17:06:21.688603+0000 mon.a (mon.0) 678 : cluster [DBG] mgrmap e19: x(active, since 2s), standbys: y 2023-07-10T17:06:22.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[28351]: cluster 2023-07-10T17:06:20.615971+0000 mgr.x (mgr.24857) 6 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:22.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[28351]: cephadm 2023-07-10T17:06:20.919360+0000 mgr.x (mgr.24857) 7 : cephadm [INF] [10/Jul/2023:17:06:20] ENGINE Bus STARTING 2023-07-10T17:06:22.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[28351]: cephadm 2023-07-10T17:06:21.070308+0000 mgr.x (mgr.24857) 8 : cephadm [INF] [10/Jul/2023:17:06:21] ENGINE Serving on http://172.21.15.192:8765 2023-07-10T17:06:22.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[28351]: cephadm 2023-07-10T17:06:21.293781+0000 mgr.x (mgr.24857) 9 : cephadm [INF] [10/Jul/2023:17:06:21] ENGINE Serving on https://172.21.15.192:7150 2023-07-10T17:06:22.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[28351]: cephadm 2023-07-10T17:06:21.294294+0000 mgr.x (mgr.24857) 10 : cephadm [INF] [10/Jul/2023:17:06:21] ENGINE Bus STARTED 2023-07-10T17:06:22.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[28351]: cluster 2023-07-10T17:06:21.575514+0000 mgr.x (mgr.24857) 11 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:22.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[28351]: audit 2023-07-10T17:06:21.680311+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-10T17:06:22.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[28351]: cluster 2023-07-10T17:06:21.680572+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2023-07-10T17:06:22.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:22 smithi116 bash[28351]: cluster 2023-07-10T17:06:21.688603+0000 mon.a (mon.0) 678 : cluster [DBG] mgrmap e19: x(active, since 2s), standbys: y 2023-07-10T17:06:24.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:23 smithi192 bash[23730]: audit 2023-07-10T17:06:22.913856+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:24.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:23 smithi192 bash[23730]: cluster 2023-07-10T17:06:23.576290+0000 mgr.x (mgr.24857) 12 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:24.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:23 smithi192 bash[23730]: cluster 2023-07-10T17:06:23.597987+0000 mon.a (mon.0) 680 : cluster [DBG] mgrmap e20: x(active, since 4s), standbys: y 2023-07-10T17:06:24.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:23 smithi116 bash[23740]: audit 2023-07-10T17:06:22.913856+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:24.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:23 smithi116 bash[23740]: cluster 2023-07-10T17:06:23.576290+0000 mgr.x (mgr.24857) 12 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:24.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:23 smithi116 bash[23740]: cluster 2023-07-10T17:06:23.597987+0000 mon.a (mon.0) 680 : cluster [DBG] mgrmap e20: x(active, since 4s), standbys: y 2023-07-10T17:06:24.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:23 smithi116 bash[28351]: audit 2023-07-10T17:06:22.913856+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:24.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:23 smithi116 bash[28351]: cluster 2023-07-10T17:06:23.576290+0000 mgr.x (mgr.24857) 12 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:24.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:23 smithi116 bash[28351]: cluster 2023-07-10T17:06:23.597987+0000 mon.a (mon.0) 680 : cluster [DBG] mgrmap e20: x(active, since 4s), standbys: y 2023-07-10T17:06:26.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:25 smithi192 bash[23730]: cluster 2023-07-10T17:06:25.577068+0000 mgr.x (mgr.24857) 13 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:26.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:25 smithi116 bash[28351]: cluster 2023-07-10T17:06:25.577068+0000 mgr.x (mgr.24857) 13 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:26.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:25 smithi116 bash[23740]: cluster 2023-07-10T17:06:25.577068+0000 mgr.x (mgr.24857) 13 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:27.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:27 smithi192 bash[23730]: audit 2023-07-10T17:06:26.493897+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:27.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:27 smithi116 bash[28351]: audit 2023-07-10T17:06:26.493897+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:27.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:27 smithi116 bash[23740]: audit 2023-07-10T17:06:26.493897+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:28.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:28 smithi192 bash[23730]: cluster 2023-07-10T17:06:27.578670+0000 mgr.x (mgr.24857) 14 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 234 KiB/s wr, 31 op/s 2023-07-10T17:06:28.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:28 smithi116 bash[28351]: cluster 2023-07-10T17:06:27.578670+0000 mgr.x (mgr.24857) 14 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 234 KiB/s wr, 31 op/s 2023-07-10T17:06:28.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:28 smithi116 bash[23740]: cluster 2023-07-10T17:06:27.578670+0000 mgr.x (mgr.24857) 14 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 234 KiB/s wr, 31 op/s 2023-07-10T17:06:30.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:30 smithi192 bash[23730]: audit 2023-07-10T17:06:28.920649+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:30.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:30 smithi192 bash[23730]: audit 2023-07-10T17:06:28.932347+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:30.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:30 smithi192 bash[23730]: audit 2023-07-10T17:06:29.154871+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:30.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:30 smithi192 bash[23730]: audit 2023-07-10T17:06:29.168354+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:30.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:30 smithi192 bash[23730]: audit 2023-07-10T17:06:29.197706+0000 mon.a (mon.0) 686 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-10T17:06:30.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:30 smithi116 bash[23740]: audit 2023-07-10T17:06:28.920649+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:30.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:30 smithi116 bash[23740]: audit 2023-07-10T17:06:28.932347+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:30.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:30 smithi116 bash[23740]: audit 2023-07-10T17:06:29.154871+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:30.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:30 smithi116 bash[23740]: audit 2023-07-10T17:06:29.168354+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:30.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:30 smithi116 bash[23740]: audit 2023-07-10T17:06:29.197706+0000 mon.a (mon.0) 686 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-10T17:06:31.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: audit 2023-07-10T17:06:28.920649+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:31.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: audit 2023-07-10T17:06:28.932347+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:31.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: audit 2023-07-10T17:06:29.154871+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:31.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: audit 2023-07-10T17:06:29.168354+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:31.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: audit 2023-07-10T17:06:29.197706+0000 mon.a (mon.0) 686 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-10T17:06:32.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:31 smithi192 bash[23730]: audit 2023-07-10T17:06:29.198844+0000 mon.b (mon.2) 47 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:06:32.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:31 smithi192 bash[23730]: audit 2023-07-10T17:06:29.199117+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:06:32.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:31 smithi192 bash[23730]: audit 2023-07-10T17:06:29.199354+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:06:32.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:31 smithi192 bash[23730]: cluster 2023-07-10T17:06:29.579478+0000 mgr.x (mgr.24857) 15 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 183 KiB/s wr, 24 op/s 2023-07-10T17:06:32.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:31 smithi192 bash[23730]: audit 2023-07-10T17:06:30.104570+0000 mon.a (mon.0) 687 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-10T17:06:32.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:31 smithi192 bash[23730]: audit 2023-07-10T17:06:30.108550+0000 mon.c (mon.1) 40 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-10T17:06:32.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:31 smithi192 bash[23730]: audit 2023-07-10T17:06:30.109834+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:06:32.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:31 smithi192 bash[23730]: audit 2023-07-10T17:06:30.110243+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:06:32.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:31 smithi192 bash[23730]: audit 2023-07-10T17:06:30.117159+0000 mon.b (mon.2) 52 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:06:32.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:31 smithi192 bash[23730]: audit 2023-07-10T17:06:30.999647+0000 mon.c (mon.1) 41 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-10T17:06:32.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:31 smithi192 bash[23730]: audit 2023-07-10T17:06:31.010580+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:32.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:31 smithi192 bash[23730]: audit 2023-07-10T17:06:31.016612+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:32.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[23740]: audit 2023-07-10T17:06:29.198844+0000 mon.b (mon.2) 47 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:06:32.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[23740]: audit 2023-07-10T17:06:29.199117+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:06:32.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[23740]: audit 2023-07-10T17:06:29.199354+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:06:32.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[23740]: cluster 2023-07-10T17:06:29.579478+0000 mgr.x (mgr.24857) 15 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 183 KiB/s wr, 24 op/s 2023-07-10T17:06:32.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[23740]: audit 2023-07-10T17:06:30.104570+0000 mon.a (mon.0) 687 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-10T17:06:32.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[23740]: audit 2023-07-10T17:06:30.108550+0000 mon.c (mon.1) 40 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-10T17:06:32.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[23740]: audit 2023-07-10T17:06:30.109834+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:06:32.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[23740]: audit 2023-07-10T17:06:30.110243+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:06:32.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[23740]: audit 2023-07-10T17:06:30.117159+0000 mon.b (mon.2) 52 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:06:32.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[23740]: audit 2023-07-10T17:06:30.999647+0000 mon.c (mon.1) 41 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-10T17:06:32.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[23740]: audit 2023-07-10T17:06:31.010580+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:32.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[23740]: audit 2023-07-10T17:06:31.016612+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:32.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: audit 2023-07-10T17:06:29.198844+0000 mon.b (mon.2) 47 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:06:32.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: audit 2023-07-10T17:06:29.199117+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:06:32.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: audit 2023-07-10T17:06:29.199354+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:06:32.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: cluster 2023-07-10T17:06:29.579478+0000 mgr.x (mgr.24857) 15 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 183 KiB/s wr, 24 op/s 2023-07-10T17:06:32.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: audit 2023-07-10T17:06:30.104570+0000 mon.a (mon.0) 687 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-10T17:06:32.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: audit 2023-07-10T17:06:30.108550+0000 mon.c (mon.1) 40 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-10T17:06:32.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: audit 2023-07-10T17:06:30.109834+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:06:32.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: audit 2023-07-10T17:06:30.110243+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:06:32.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: audit 2023-07-10T17:06:30.117159+0000 mon.b (mon.2) 52 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:06:32.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: audit 2023-07-10T17:06:30.999647+0000 mon.c (mon.1) 41 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-10T17:06:32.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: audit 2023-07-10T17:06:31.010580+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:32.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:31 smithi116 bash[28351]: audit 2023-07-10T17:06:31.016612+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:33.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:32 smithi192 bash[23730]: audit 2023-07-10T17:06:31.022521+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:06:33.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:32 smithi192 bash[23730]: audit 2023-07-10T17:06:31.022653+0000 mon.b (mon.2) 54 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-10T17:06:33.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:32 smithi192 bash[23730]: cluster 2023-07-10T17:06:31.581124+0000 mgr.x (mgr.24857) 16 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 215 KiB/s wr, 25 op/s 2023-07-10T17:06:33.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:32 smithi192 bash[23730]: audit 2023-07-10T17:06:31.918324+0000 mon.b (mon.2) 55 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-10T17:06:33.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:32 smithi192 bash[23730]: audit 2023-07-10T17:06:31.920373+0000 mon.b (mon.2) 56 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:06:33.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:32 smithi192 bash[23730]: audit 2023-07-10T17:06:31.920763+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:06:33.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:32 smithi192 bash[23730]: audit 2023-07-10T17:06:31.924641+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:33.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:32 smithi192 bash[23730]: audit 2023-07-10T17:06:31.944139+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:33.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:32 smithi116 bash[23740]: audit 2023-07-10T17:06:31.022521+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:06:33.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:32 smithi116 bash[23740]: audit 2023-07-10T17:06:31.022653+0000 mon.b (mon.2) 54 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-10T17:06:33.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:32 smithi116 bash[23740]: cluster 2023-07-10T17:06:31.581124+0000 mgr.x (mgr.24857) 16 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 215 KiB/s wr, 25 op/s 2023-07-10T17:06:33.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:32 smithi116 bash[23740]: audit 2023-07-10T17:06:31.918324+0000 mon.b (mon.2) 55 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-10T17:06:33.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:32 smithi116 bash[23740]: audit 2023-07-10T17:06:31.920373+0000 mon.b (mon.2) 56 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:06:33.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:32 smithi116 bash[23740]: audit 2023-07-10T17:06:31.920763+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:06:33.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:32 smithi116 bash[23740]: audit 2023-07-10T17:06:31.924641+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:33.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:32 smithi116 bash[23740]: audit 2023-07-10T17:06:31.944139+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:33.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:32 smithi116 bash[28351]: audit 2023-07-10T17:06:31.022521+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:06:33.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:32 smithi116 bash[28351]: audit 2023-07-10T17:06:31.022653+0000 mon.b (mon.2) 54 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-10T17:06:33.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:32 smithi116 bash[28351]: cluster 2023-07-10T17:06:31.581124+0000 mgr.x (mgr.24857) 16 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 215 KiB/s wr, 25 op/s 2023-07-10T17:06:33.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:32 smithi116 bash[28351]: audit 2023-07-10T17:06:31.918324+0000 mon.b (mon.2) 55 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-10T17:06:33.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:32 smithi116 bash[28351]: audit 2023-07-10T17:06:31.920373+0000 mon.b (mon.2) 56 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:06:33.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:32 smithi116 bash[28351]: audit 2023-07-10T17:06:31.920763+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:06:33.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:32 smithi116 bash[28351]: audit 2023-07-10T17:06:31.924641+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:33.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:32 smithi116 bash[28351]: audit 2023-07-10T17:06:31.944139+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:34.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:34 smithi192 bash[23730]: cluster 2023-07-10T17:06:33.581914+0000 mgr.x (mgr.24857) 17 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 181 KiB/s wr, 21 op/s 2023-07-10T17:06:34.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:34 smithi116 bash[23740]: cluster 2023-07-10T17:06:33.581914+0000 mgr.x (mgr.24857) 17 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 181 KiB/s wr, 21 op/s 2023-07-10T17:06:34.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:34 smithi116 bash[28351]: cluster 2023-07-10T17:06:33.581914+0000 mgr.x (mgr.24857) 17 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 181 KiB/s wr, 21 op/s 2023-07-10T17:06:36.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:35 smithi192 bash[23730]: cluster 2023-07-10T17:06:35.582690+0000 mgr.x (mgr.24857) 18 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 179 KiB/s wr, 20 op/s 2023-07-10T17:06:36.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:35 smithi116 bash[23740]: cluster 2023-07-10T17:06:35.582690+0000 mgr.x (mgr.24857) 18 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 179 KiB/s wr, 20 op/s 2023-07-10T17:06:36.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:35 smithi116 bash[28351]: cluster 2023-07-10T17:06:35.582690+0000 mgr.x (mgr.24857) 18 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 179 KiB/s wr, 20 op/s 2023-07-10T17:06:37.976 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:37 smithi192 bash[23730]: cluster 2023-07-10T17:06:37.584670+0000 mgr.x (mgr.24857) 19 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.3 KiB/s rd, 201 KiB/s wr, 22 op/s 2023-07-10T17:06:37.991 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:37 smithi116 bash[28351]: cluster 2023-07-10T17:06:37.584670+0000 mgr.x (mgr.24857) 19 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.3 KiB/s rd, 201 KiB/s wr, 22 op/s 2023-07-10T17:06:37.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:37 smithi116 bash[23740]: cluster 2023-07-10T17:06:37.584670+0000 mgr.x (mgr.24857) 19 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.3 KiB/s rd, 201 KiB/s wr, 22 op/s 2023-07-10T17:06:39.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:39 smithi192 bash[23730]: cluster 2023-07-10T17:06:39.585473+0000 mgr.x (mgr.24857) 20 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-07-10T17:06:40.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:39 smithi116 bash[28351]: cluster 2023-07-10T17:06:39.585473+0000 mgr.x (mgr.24857) 20 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-07-10T17:06:40.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:39 smithi116 bash[23740]: cluster 2023-07-10T17:06:39.585473+0000 mgr.x (mgr.24857) 20 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-07-10T17:06:42.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:41 smithi192 bash[23730]: cluster 2023-07-10T17:06:41.587179+0000 mgr.x (mgr.24857) 21 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-07-10T17:06:42.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:41 smithi116 bash[23740]: cluster 2023-07-10T17:06:41.587179+0000 mgr.x (mgr.24857) 21 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-07-10T17:06:42.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:41 smithi116 bash[28351]: cluster 2023-07-10T17:06:41.587179+0000 mgr.x (mgr.24857) 21 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-07-10T17:06:42.548 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:06:42.697 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:42 smithi116 systemd[1]: Stopping Ceph node-exporter.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:06:43.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[23740]: audit 2023-07-10T17:06:41.926259+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[23740]: audit 2023-07-10T17:06:41.936940+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[23740]: audit 2023-07-10T17:06:41.946583+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[23740]: audit 2023-07-10T17:06:41.958575+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[23740]: cluster 2023-07-10T17:06:41.965050+0000 mgr.x (mgr.24857) 22 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 887 B/s rd, 25 KiB/s wr, 2 op/s 2023-07-10T17:06:43.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[23740]: audit 2023-07-10T17:06:41.971579+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[23740]: audit 2023-07-10T17:06:41.983786+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[23740]: audit 2023-07-10T17:06:41.997947+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[23740]: audit 2023-07-10T17:06:42.010796+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[23740]: audit 2023-07-10T17:06:42.023567+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[23740]: cephadm 2023-07-10T17:06:42.036589+0000 mgr.x (mgr.24857) 23 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2023-07-10T17:06:43.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[23740]: cephadm 2023-07-10T17:06:42.038121+0000 mgr.x (mgr.24857) 24 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2023-07-10T17:06:43.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[23740]: audit 2023-07-10T17:06:42.046447+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[23740]: cephadm 2023-07-10T17:06:42.160726+0000 mgr.x (mgr.24857) 25 : cephadm [INF] Reconfiguring node-exporter.a (dependencies changed)... 2023-07-10T17:06:43.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[23740]: cephadm 2023-07-10T17:06:42.161197+0000 mgr.x (mgr.24857) 26 : cephadm [INF] Reconfiguring daemon node-exporter.a on smithi116 2023-07-10T17:06:43.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[23740]: audit 2023-07-10T17:06:42.530370+0000 mgr.x (mgr.24857) 27 : audit [DBG] from='client.24943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:43.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[28351]: audit 2023-07-10T17:06:41.926259+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[28351]: audit 2023-07-10T17:06:41.936940+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[28351]: audit 2023-07-10T17:06:41.946583+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[28351]: audit 2023-07-10T17:06:41.958575+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[28351]: cluster 2023-07-10T17:06:41.965050+0000 mgr.x (mgr.24857) 22 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 887 B/s rd, 25 KiB/s wr, 2 op/s 2023-07-10T17:06:43.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[28351]: audit 2023-07-10T17:06:41.971579+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[28351]: audit 2023-07-10T17:06:41.983786+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[28351]: audit 2023-07-10T17:06:41.997947+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[28351]: audit 2023-07-10T17:06:42.010796+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[28351]: audit 2023-07-10T17:06:42.023567+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[28351]: cephadm 2023-07-10T17:06:42.036589+0000 mgr.x (mgr.24857) 23 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2023-07-10T17:06:43.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[28351]: cephadm 2023-07-10T17:06:42.038121+0000 mgr.x (mgr.24857) 24 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2023-07-10T17:06:43.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[28351]: audit 2023-07-10T17:06:42.046447+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[28351]: cephadm 2023-07-10T17:06:42.160726+0000 mgr.x (mgr.24857) 25 : cephadm [INF] Reconfiguring node-exporter.a (dependencies changed)... 2023-07-10T17:06:43.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[28351]: cephadm 2023-07-10T17:06:42.161197+0000 mgr.x (mgr.24857) 26 : cephadm [INF] Reconfiguring daemon node-exporter.a on smithi116 2023-07-10T17:06:43.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:42 smithi116 bash[28351]: audit 2023-07-10T17:06:42.530370+0000 mgr.x (mgr.24857) 27 : audit [DBG] from='client.24943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:43.320 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:06:43.320 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (5m) 14s ago 6m 10.5M - 0.20.0 0881eb8f169f 54f60b4d9c19 2023-07-10T17:06:43.321 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (5m) 14s ago 5m 26.5M - 6.7.4 557c83e11646 ca7910fb45c5 2023-07-10T17:06:43.321 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 error 14s ago 3m - - 2023-07-10T17:06:43.321 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443 running (33s) 14s ago 10m 460M - 18.1.2-30-g8507faa7 17e8f0d79247 d22a6b7f47df 2023-07-10T17:06:43.321 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:9283 running (12m) 14s ago 12m 354M - 16.2.0 24ecd6d5f14c 9776722e922e 2023-07-10T17:06:43.321 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (12m) 14s ago 12m 58.6M 2048M 16.2.0 24ecd6d5f14c 33f26d450a67 2023-07-10T17:06:43.321 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (10m) 14s ago 10m 52.3M 2048M 16.2.0 24ecd6d5f14c 9ad4329e5488 2023-07-10T17:06:43.321 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (10m) 14s ago 10m 48.4M 2048M 16.2.0 24ecd6d5f14c 31e3ac12741b 2023-07-10T17:06:43.322 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 starting - - - - 2023-07-10T17:06:43.322 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (6m) 14s ago 6m 11.1M - 0.18.1 e5a616e4b9cf db9b56cbd267 2023-07-10T17:06:43.322 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (10m) 14s ago 10m 52.8M 4096M 16.2.0 24ecd6d5f14c 4fa26367c830 2023-07-10T17:06:43.322 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (9m) 14s ago 9m 52.0M 4096M 16.2.0 24ecd6d5f14c c157509b3fd4 2023-07-10T17:06:43.322 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (9m) 14s ago 9m 48.1M 4096M 16.2.0 24ecd6d5f14c f1fd77b3c454 2023-07-10T17:06:43.322 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (8m) 14s ago 8m 53.4M 4096M 16.2.0 24ecd6d5f14c 60949d479c11 2023-07-10T17:06:43.322 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (8m) 14s ago 8m 50.9M 4096M 16.2.0 24ecd6d5f14c 75fc0b36aab8 2023-07-10T17:06:43.323 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (7m) 14s ago 8m 49.2M 4096M 16.2.0 24ecd6d5f14c d673065333dd 2023-07-10T17:06:43.323 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (7m) 14s ago 7m 49.4M 4096M 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:06:43.323 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (7m) 14s ago 7m 52.7M 4096M 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:06:43.323 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 running (5m) 14s ago 6m 25.5M - 2.18.1 de242295e225 95c5a7e6f5c5 2023-07-10T17:06:43.323 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (4m) 14s ago 4m 52.8M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:06:43.323 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (4m) 14s ago 4m 52.6M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:06:43.323 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (3m) 14s ago 3m 53.0M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:06:43.324 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (3m) 14s ago 3m 54.6M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:06:43.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:42 smithi192 bash[23730]: audit 2023-07-10T17:06:41.926259+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:42 smithi192 bash[23730]: audit 2023-07-10T17:06:41.936940+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:42 smithi192 bash[23730]: audit 2023-07-10T17:06:41.946583+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:42 smithi192 bash[23730]: audit 2023-07-10T17:06:41.958575+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:42 smithi192 bash[23730]: cluster 2023-07-10T17:06:41.965050+0000 mgr.x (mgr.24857) 22 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 887 B/s rd, 25 KiB/s wr, 2 op/s 2023-07-10T17:06:43.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:42 smithi192 bash[23730]: audit 2023-07-10T17:06:41.971579+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:42 smithi192 bash[23730]: audit 2023-07-10T17:06:41.983786+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:42 smithi192 bash[23730]: audit 2023-07-10T17:06:41.997947+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:42 smithi192 bash[23730]: audit 2023-07-10T17:06:42.010796+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:42 smithi192 bash[23730]: audit 2023-07-10T17:06:42.023567+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:42 smithi192 bash[23730]: cephadm 2023-07-10T17:06:42.036589+0000 mgr.x (mgr.24857) 23 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2023-07-10T17:06:43.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:42 smithi192 bash[23730]: cephadm 2023-07-10T17:06:42.038121+0000 mgr.x (mgr.24857) 24 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2023-07-10T17:06:43.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:42 smithi192 bash[23730]: audit 2023-07-10T17:06:42.046447+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:43.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:42 smithi192 bash[23730]: cephadm 2023-07-10T17:06:42.160726+0000 mgr.x (mgr.24857) 25 : cephadm [INF] Reconfiguring node-exporter.a (dependencies changed)... 2023-07-10T17:06:43.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:42 smithi192 bash[23730]: cephadm 2023-07-10T17:06:42.161197+0000 mgr.x (mgr.24857) 26 : cephadm [INF] Reconfiguring daemon node-exporter.a on smithi116 2023-07-10T17:06:43.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:42 smithi192 bash[23730]: audit 2023-07-10T17:06:42.530370+0000 mgr.x (mgr.24857) 27 : audit [DBG] from='client.24943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:43.643 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:43 smithi116 docker[56493]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-node-exporter.a 2023-07-10T17:06:43.643 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:43 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2023-07-10T17:06:43.643 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:43 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@node-exporter.a.service: Failed with result 'exit-code'. 2023-07-10T17:06:43.643 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:43 smithi116 systemd[1]: Stopped Ceph node-exporter.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:06:43.643 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:43 smithi116 systemd[1]: Started Ceph node-exporter.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:06:43.808 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:06:43.809 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:06:43.809 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2023-07-10T17:06:43.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:06:43.809 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:06:43.809 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2023-07-10T17:06:43.809 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 1 2023-07-10T17:06:43.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:06:43.810 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:06:43.810 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-10T17:06:43.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:06:43.810 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": {}, 2023-07-10T17:06:43.810 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:06:43.810 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:06:43.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:06:43.810 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:06:43.811 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 16, 2023-07-10T17:06:43.811 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 1 2023-07-10T17:06:43.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:06:43.811 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:06:44.177 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:06:44.177 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:06:44.177 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:06:44.178 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:06:44.178 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [], 2023-07-10T17:06:44.178 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "1/23 daemons upgraded", 2023-07-10T17:06:44.178 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "", 2023-07-10T17:06:44.178 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:06:44.178 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:06:44.292 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:43 smithi116 bash[28351]: audit 2023-07-10T17:06:42.901817+0000 mgr.x (mgr.24857) 28 : audit [DBG] from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:44.292 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:43 smithi116 bash[28351]: cluster 2023-07-10T17:06:42.954906+0000 mon.a (mon.0) 702 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:06:44.292 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:43 smithi116 bash[28351]: audit 2023-07-10T17:06:43.254997+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:44.293 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:43 smithi116 bash[28351]: audit 2023-07-10T17:06:43.262862+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:44.293 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:43 smithi116 bash[28351]: cephadm 2023-07-10T17:06:43.279509+0000 mgr.x (mgr.24857) 29 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-07-10T17:06:44.293 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:43 smithi116 bash[28351]: audit 2023-07-10T17:06:43.282009+0000 mgr.x (mgr.24857) 30 : audit [DBG] from='client.24955 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:44.293 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:43 smithi116 bash[28351]: cephadm 2023-07-10T17:06:43.317778+0000 mgr.x (mgr.24857) 31 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi116 2023-07-10T17:06:44.293 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:43 smithi116 bash[28351]: audit 2023-07-10T17:06:43.804795+0000 mon.a (mon.0) 705 : audit [DBG] from='client.? 172.21.15.116:0/1788717718' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:06:44.293 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:43 smithi116 bash[23740]: audit 2023-07-10T17:06:42.901817+0000 mgr.x (mgr.24857) 28 : audit [DBG] from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:44.294 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:43 smithi116 bash[23740]: cluster 2023-07-10T17:06:42.954906+0000 mon.a (mon.0) 702 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:06:44.294 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:43 smithi116 bash[23740]: audit 2023-07-10T17:06:43.254997+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:44.294 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:43 smithi116 bash[23740]: audit 2023-07-10T17:06:43.262862+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:44.294 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:43 smithi116 bash[23740]: cephadm 2023-07-10T17:06:43.279509+0000 mgr.x (mgr.24857) 29 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-07-10T17:06:44.294 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:43 smithi116 bash[23740]: audit 2023-07-10T17:06:43.282009+0000 mgr.x (mgr.24857) 30 : audit [DBG] from='client.24955 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:44.294 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:43 smithi116 bash[23740]: cephadm 2023-07-10T17:06:43.317778+0000 mgr.x (mgr.24857) 31 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi116 2023-07-10T17:06:44.294 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:43 smithi116 bash[23740]: audit 2023-07-10T17:06:43.804795+0000 mon.a (mon.0) 705 : audit [DBG] from='client.? 172.21.15.116:0/1788717718' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:06:44.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:43 smithi192 bash[23730]: audit 2023-07-10T17:06:42.901817+0000 mgr.x (mgr.24857) 28 : audit [DBG] from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:44.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:43 smithi192 bash[23730]: cluster 2023-07-10T17:06:42.954906+0000 mon.a (mon.0) 702 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:06:44.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:43 smithi192 bash[23730]: audit 2023-07-10T17:06:43.254997+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:44.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:43 smithi192 bash[23730]: audit 2023-07-10T17:06:43.262862+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:44.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:43 smithi192 bash[23730]: cephadm 2023-07-10T17:06:43.279509+0000 mgr.x (mgr.24857) 29 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-07-10T17:06:44.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:43 smithi192 bash[23730]: audit 2023-07-10T17:06:43.282009+0000 mgr.x (mgr.24857) 30 : audit [DBG] from='client.24955 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:44.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:43 smithi192 bash[23730]: cephadm 2023-07-10T17:06:43.317778+0000 mgr.x (mgr.24857) 31 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi116 2023-07-10T17:06:44.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:43 smithi192 bash[23730]: audit 2023-07-10T17:06:43.804795+0000 mon.a (mon.0) 705 : audit [DBG] from='client.? 172.21.15.116:0/1788717718' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:06:44.615 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2023-07-10T17:06:44.616 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2023-07-10T17:06:44.616 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:06:44.643 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2023-07-10T17:06:44.644 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2023-07-10T17:06:44.644 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2023-07-10T17:06:44.644 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - arp" source="node_exporter.go:104" 2023-07-10T17:06:44.644 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - bcache" source="node_exporter.go:104" 2023-07-10T17:06:44.644 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - bonding" source="node_exporter.go:104" 2023-07-10T17:06:44.644 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2023-07-10T17:06:44.645 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - cpu" source="node_exporter.go:104" 2023-07-10T17:06:44.645 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2023-07-10T17:06:44.645 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2023-07-10T17:06:44.645 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - edac" source="node_exporter.go:104" 2023-07-10T17:06:44.645 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - entropy" source="node_exporter.go:104" 2023-07-10T17:06:44.645 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - filefd" source="node_exporter.go:104" 2023-07-10T17:06:44.645 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2023-07-10T17:06:44.645 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2023-07-10T17:06:44.646 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2023-07-10T17:06:44.646 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2023-07-10T17:06:44.646 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2023-07-10T17:06:44.646 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2023-07-10T17:06:44.646 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2023-07-10T17:06:44.646 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - netclass" source="node_exporter.go:104" 2023-07-10T17:06:44.646 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - netdev" source="node_exporter.go:104" 2023-07-10T17:06:44.646 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - netstat" source="node_exporter.go:104" 2023-07-10T17:06:44.647 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - nfs" source="node_exporter.go:104" 2023-07-10T17:06:44.647 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2023-07-10T17:06:44.647 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - pressure" source="node_exporter.go:104" 2023-07-10T17:06:44.647 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2023-07-10T17:06:44.647 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - stat" source="node_exporter.go:104" 2023-07-10T17:06:44.647 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - textfile" source="node_exporter.go:104" 2023-07-10T17:06:44.647 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - time" source="node_exporter.go:104" 2023-07-10T17:06:44.648 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - uname" source="node_exporter.go:104" 2023-07-10T17:06:44.648 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2023-07-10T17:06:44.648 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - xfs" source="node_exporter.go:104" 2023-07-10T17:06:44.648 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg=" - zfs" source="node_exporter.go:104" 2023-07-10T17:06:44.648 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[56593]: time="2023-07-10T17:06:44Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2023-07-10T17:06:45.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:44 smithi192 bash[23730]: cluster 2023-07-10T17:06:43.965895+0000 mgr.x (mgr.24857) 32 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 887 B/s rd, 25 KiB/s wr, 2 op/s 2023-07-10T17:06:45.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:44 smithi192 bash[23730]: audit 2023-07-10T17:06:44.173725+0000 mgr.x (mgr.24857) 33 : audit [DBG] from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:45.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:44 smithi192 bash[23730]: audit 2023-07-10T17:06:44.612174+0000 mon.c (mon.1) 42 : audit [DBG] from='client.? 172.21.15.116:0/3195580286' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:06:45.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[23740]: cluster 2023-07-10T17:06:43.965895+0000 mgr.x (mgr.24857) 32 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 887 B/s rd, 25 KiB/s wr, 2 op/s 2023-07-10T17:06:45.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[23740]: audit 2023-07-10T17:06:44.173725+0000 mgr.x (mgr.24857) 33 : audit [DBG] from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:45.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[23740]: audit 2023-07-10T17:06:44.612174+0000 mon.c (mon.1) 42 : audit [DBG] from='client.? 172.21.15.116:0/3195580286' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:06:45.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[28351]: cluster 2023-07-10T17:06:43.965895+0000 mgr.x (mgr.24857) 32 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 887 B/s rd, 25 KiB/s wr, 2 op/s 2023-07-10T17:06:45.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[28351]: audit 2023-07-10T17:06:44.173725+0000 mgr.x (mgr.24857) 33 : audit [DBG] from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:06:45.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:44 smithi116 bash[28351]: audit 2023-07-10T17:06:44.612174+0000 mon.c (mon.1) 42 : audit [DBG] from='client.? 172.21.15.116:0/3195580286' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:06:47.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:47 smithi192 bash[23730]: cluster 2023-07-10T17:06:45.967582+0000 mgr.x (mgr.24857) 34 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 887 B/s rd, 25 KiB/s wr, 2 op/s 2023-07-10T17:06:47.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:47 smithi116 bash[23740]: cluster 2023-07-10T17:06:45.967582+0000 mgr.x (mgr.24857) 34 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 887 B/s rd, 25 KiB/s wr, 2 op/s 2023-07-10T17:06:47.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:47 smithi116 bash[28351]: cluster 2023-07-10T17:06:45.967582+0000 mgr.x (mgr.24857) 34 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 887 B/s rd, 25 KiB/s wr, 2 op/s 2023-07-10T17:06:49.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:49 smithi192 bash[23730]: cluster 2023-07-10T17:06:47.968343+0000 mgr.x (mgr.24857) 35 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:49.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:49 smithi116 bash[23740]: cluster 2023-07-10T17:06:47.968343+0000 mgr.x (mgr.24857) 35 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:49.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:49 smithi116 bash[28351]: cluster 2023-07-10T17:06:47.968343+0000 mgr.x (mgr.24857) 35 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:51.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:51 smithi192 bash[23730]: cluster 2023-07-10T17:06:49.969107+0000 mgr.x (mgr.24857) 36 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:51.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:51 smithi116 bash[28351]: cluster 2023-07-10T17:06:49.969107+0000 mgr.x (mgr.24857) 36 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:51.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:51 smithi116 bash[23740]: cluster 2023-07-10T17:06:49.969107+0000 mgr.x (mgr.24857) 36 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:52.304 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:06:52 smithi116 systemd[1]: Stopping Ceph alertmanager.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:06:52.304 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:06:52 smithi116 bash[44891]: level=info ts=2023-07-10T17:06:52.126Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2023-07-10T17:06:53.143 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:06:52 smithi116 docker[57008]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-alertmanager.a 2023-07-10T17:06:53.143 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:06:52 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@alertmanager.a.service: Succeeded. 2023-07-10T17:06:53.143 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:06:52 smithi116 systemd[1]: Stopped Ceph alertmanager.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:06:53.143 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:06:52 smithi116 systemd[1]: Started Ceph alertmanager.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:06:53.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:53 smithi116 bash[28351]: cluster 2023-07-10T17:06:51.970842+0000 mgr.x (mgr.24857) 37 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:53.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:53 smithi116 bash[28351]: audit 2023-07-10T17:06:52.897319+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:53.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:53 smithi116 bash[28351]: audit 2023-07-10T17:06:52.909687+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:53.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:53 smithi116 bash[28351]: audit 2023-07-10T17:06:52.954675+0000 mon.b (mon.2) 58 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:06:53.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:53 smithi116 bash[28351]: audit 2023-07-10T17:06:52.954930+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:06:53.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:53 smithi116 bash[28351]: audit 2023-07-10T17:06:52.967662+0000 mon.b (mon.2) 59 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:06:53.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:53 smithi116 bash[23740]: cluster 2023-07-10T17:06:51.970842+0000 mgr.x (mgr.24857) 37 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:53.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:53 smithi116 bash[23740]: audit 2023-07-10T17:06:52.897319+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:53.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:53 smithi116 bash[23740]: audit 2023-07-10T17:06:52.909687+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:53.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:53 smithi116 bash[23740]: audit 2023-07-10T17:06:52.954675+0000 mon.b (mon.2) 58 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:06:53.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:53 smithi116 bash[23740]: audit 2023-07-10T17:06:52.954930+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:06:53.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:53 smithi116 bash[23740]: audit 2023-07-10T17:06:52.967662+0000 mon.b (mon.2) 59 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:06:53.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:53 smithi192 bash[23730]: cluster 2023-07-10T17:06:51.970842+0000 mgr.x (mgr.24857) 37 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:53.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:53 smithi192 bash[23730]: audit 2023-07-10T17:06:52.897319+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:53.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:53 smithi192 bash[23730]: audit 2023-07-10T17:06:52.909687+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:53.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:53 smithi192 bash[23730]: audit 2023-07-10T17:06:52.954675+0000 mon.b (mon.2) 58 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:06:53.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:53 smithi192 bash[23730]: audit 2023-07-10T17:06:52.954930+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:06:53.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:53 smithi192 bash[23730]: audit 2023-07-10T17:06:52.967662+0000 mon.b (mon.2) 59 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:06:54.840 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:54 smithi116 bash[23740]: cephadm 2023-07-10T17:06:52.953889+0000 mgr.x (mgr.24857) 38 : cephadm [INF] Reconfiguring iscsi.foo.smithi116.fkthyl (dependencies changed)... 2023-07-10T17:06:54.841 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:54 smithi116 bash[23740]: cephadm 2023-07-10T17:06:52.969207+0000 mgr.x (mgr.24857) 39 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:06:54.841 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:54 smithi116 bash[28351]: cephadm 2023-07-10T17:06:52.953889+0000 mgr.x (mgr.24857) 38 : cephadm [INF] Reconfiguring iscsi.foo.smithi116.fkthyl (dependencies changed)... 2023-07-10T17:06:54.841 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:54 smithi116 bash[28351]: cephadm 2023-07-10T17:06:52.969207+0000 mgr.x (mgr.24857) 39 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:06:54.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:54 smithi192 bash[23730]: cephadm 2023-07-10T17:06:52.953889+0000 mgr.x (mgr.24857) 38 : cephadm [INF] Reconfiguring iscsi.foo.smithi116.fkthyl (dependencies changed)... 2023-07-10T17:06:54.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:54 smithi192 bash[23730]: cephadm 2023-07-10T17:06:52.969207+0000 mgr.x (mgr.24857) 39 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:06:55.143 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:06:54 smithi116 bash[57058]: level=info ts=2023-07-10T17:06:54.905Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2023-07-10T17:06:55.143 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:06:54 smithi116 bash[57058]: level=info ts=2023-07-10T17:06:54.905Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2023-07-10T17:06:55.143 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:06:54 smithi116 bash[57058]: level=info ts=2023-07-10T17:06:54.907Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.116 port=9094 2023-07-10T17:06:55.143 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:06:54 smithi116 bash[57058]: level=info ts=2023-07-10T17:06:54.908Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2023-07-10T17:06:55.144 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:06:54 smithi116 bash[57058]: level=info ts=2023-07-10T17:06:54.941Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-10T17:06:55.144 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:06:54 smithi116 bash[57058]: level=info ts=2023-07-10T17:06:54.942Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-10T17:06:55.144 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:06:54 smithi116 bash[57058]: level=info ts=2023-07-10T17:06:54.947Z caller=main.go:497 msg=Listening address=:9093 2023-07-10T17:06:55.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:55 smithi192 bash[23730]: cluster 2023-07-10T17:06:53.971791+0000 mgr.x (mgr.24857) 40 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:55.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:55 smithi116 bash[23740]: cluster 2023-07-10T17:06:53.971791+0000 mgr.x (mgr.24857) 40 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:55.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:55 smithi116 bash[28351]: cluster 2023-07-10T17:06:53.971791+0000 mgr.x (mgr.24857) 40 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:57.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:56 smithi192 bash[23730]: audit 2023-07-10T17:06:55.747092+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:57.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:56 smithi192 bash[23730]: audit 2023-07-10T17:06:55.760640+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:57.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:56 smithi192 bash[23730]: cephadm 2023-07-10T17:06:55.793158+0000 mgr.x (mgr.24857) 41 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-10T17:06:57.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:56 smithi192 bash[23730]: cluster 2023-07-10T17:06:55.973510+0000 mgr.x (mgr.24857) 42 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:57.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:56 smithi192 bash[23730]: cephadm 2023-07-10T17:06:56.037644+0000 mgr.x (mgr.24857) 43 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi192 2023-07-10T17:06:57.143 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:06:56 smithi116 bash[57058]: level=info ts=2023-07-10T17:06:56.909Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000164318s 2023-07-10T17:06:57.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:56 smithi116 bash[23740]: audit 2023-07-10T17:06:55.747092+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:57.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:56 smithi116 bash[23740]: audit 2023-07-10T17:06:55.760640+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:57.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:56 smithi116 bash[23740]: cephadm 2023-07-10T17:06:55.793158+0000 mgr.x (mgr.24857) 41 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-10T17:06:57.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:56 smithi116 bash[23740]: cluster 2023-07-10T17:06:55.973510+0000 mgr.x (mgr.24857) 42 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:57.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:56 smithi116 bash[23740]: cephadm 2023-07-10T17:06:56.037644+0000 mgr.x (mgr.24857) 43 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi192 2023-07-10T17:06:57.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:56 smithi116 bash[28351]: audit 2023-07-10T17:06:55.747092+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:57.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:56 smithi116 bash[28351]: audit 2023-07-10T17:06:55.760640+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:06:57.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:56 smithi116 bash[28351]: cephadm 2023-07-10T17:06:55.793158+0000 mgr.x (mgr.24857) 41 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-10T17:06:57.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:56 smithi116 bash[28351]: cluster 2023-07-10T17:06:55.973510+0000 mgr.x (mgr.24857) 42 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:57.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:56 smithi116 bash[28351]: cephadm 2023-07-10T17:06:56.037644+0000 mgr.x (mgr.24857) 43 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi192 2023-07-10T17:06:58.312 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:57 smithi116 bash[23740]: audit 2023-07-10T17:06:57.933232+0000 mon.a (mon.0) 711 : audit [DBG] from='client.? 172.21.15.116:0/696330846' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:06:58.313 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:57 smithi116 bash[28351]: audit 2023-07-10T17:06:57.933232+0000 mon.a (mon.0) 711 : audit [DBG] from='client.? 172.21.15.116:0/696330846' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:06:58.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:57 smithi192 bash[23730]: audit 2023-07-10T17:06:57.933232+0000 mon.a (mon.0) 711 : audit [DBG] from='client.? 172.21.15.116:0/696330846' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:06:59.369 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:59 smithi116 bash[28351]: cluster 2023-07-10T17:06:57.974263+0000 mgr.x (mgr.24857) 44 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:59.369 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:06:59 smithi116 bash[28351]: audit 2023-07-10T17:06:58.303146+0000 mon.c (mon.1) 43 : audit [INF] from='client.? 172.21.15.116:0/1904757304' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: access denied 2023-07-10T17:06:59.370 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:59 smithi116 bash[23740]: cluster 2023-07-10T17:06:57.974263+0000 mgr.x (mgr.24857) 44 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:59.370 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:06:59 smithi116 bash[23740]: audit 2023-07-10T17:06:58.303146+0000 mon.c (mon.1) 43 : audit [INF] from='client.? 172.21.15.116:0/1904757304' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: access denied 2023-07-10T17:06:59.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:59 smithi192 bash[23730]: cluster 2023-07-10T17:06:57.974263+0000 mgr.x (mgr.24857) 44 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:06:59.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:06:59 smithi192 bash[23730]: audit 2023-07-10T17:06:58.303146+0000 mon.c (mon.1) 43 : audit [INF] from='client.? 172.21.15.116:0/1904757304' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: access denied 2023-07-10T17:07:01.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:01 smithi192 bash[23730]: cluster 2023-07-10T17:06:59.974943+0000 mgr.x (mgr.24857) 45 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:01.392 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:01 smithi116 bash[28351]: cluster 2023-07-10T17:06:59.974943+0000 mgr.x (mgr.24857) 45 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:01.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:01 smithi116 bash[23740]: cluster 2023-07-10T17:06:59.974943+0000 mgr.x (mgr.24857) 45 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:03.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:03 smithi192 bash[23730]: cluster 2023-07-10T17:07:01.976574+0000 mgr.x (mgr.24857) 46 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:03.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:03 smithi116 bash[23740]: cluster 2023-07-10T17:07:01.976574+0000 mgr.x (mgr.24857) 46 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:03.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:03 smithi116 bash[28351]: cluster 2023-07-10T17:07:01.976574+0000 mgr.x (mgr.24857) 46 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:05.393 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:07:04 smithi116 bash[57058]: level=info ts=2023-07-10T17:07:04.910Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.001147631s 2023-07-10T17:07:05.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:05 smithi192 bash[23730]: cluster 2023-07-10T17:07:03.977376+0000 mgr.x (mgr.24857) 47 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:05.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:05 smithi116 bash[23740]: cluster 2023-07-10T17:07:03.977376+0000 mgr.x (mgr.24857) 47 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:05.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:05 smithi116 bash[28351]: cluster 2023-07-10T17:07:03.977376+0000 mgr.x (mgr.24857) 47 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:07.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:07 smithi192 bash[23730]: cluster 2023-07-10T17:07:05.979103+0000 mgr.x (mgr.24857) 48 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:07.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:07 smithi116 bash[23740]: cluster 2023-07-10T17:07:05.979103+0000 mgr.x (mgr.24857) 48 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:07.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:07 smithi116 bash[28351]: cluster 2023-07-10T17:07:05.979103+0000 mgr.x (mgr.24857) 48 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:09.576 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 systemd[1]: Stopping Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:07:09.576 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 bash[38545]: level=warn ts=2023-07-10T17:07:09.273Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..." 2023-07-10T17:07:09.576 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 bash[38545]: level=info ts=2023-07-10T17:07:09.273Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-10T17:07:09.576 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 bash[38545]: level=info ts=2023-07-10T17:07:09.273Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-10T17:07:09.576 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 bash[38545]: level=info ts=2023-07-10T17:07:09.273Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-10T17:07:09.577 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 bash[38545]: level=info ts=2023-07-10T17:07:09.273Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-10T17:07:09.577 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 bash[38545]: level=info ts=2023-07-10T17:07:09.274Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-10T17:07:09.577 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 bash[38545]: level=info ts=2023-07-10T17:07:09.274Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-10T17:07:09.577 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 bash[38545]: level=info ts=2023-07-10T17:07:09.274Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-10T17:07:09.577 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 bash[38545]: level=info ts=2023-07-10T17:07:09.274Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-10T17:07:09.577 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 bash[38545]: level=info ts=2023-07-10T17:07:09.275Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-10T17:07:09.577 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 bash[38545]: level=info ts=2023-07-10T17:07:09.275Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-10T17:07:09.577 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 bash[38545]: level=info ts=2023-07-10T17:07:09.275Z caller=main.go:761 msg="See you next time!" 2023-07-10T17:07:09.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:09 smithi192 bash[23730]: cluster 2023-07-10T17:07:07.979998+0000 mgr.x (mgr.24857) 49 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:09.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:09 smithi116 bash[28351]: cluster 2023-07-10T17:07:07.979998+0000 mgr.x (mgr.24857) 49 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:09.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:09 smithi116 bash[23740]: cluster 2023-07-10T17:07:07.979998+0000 mgr.x (mgr.24857) 49 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:10.368 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 docker[44347]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-prometheus.a 2023-07-10T17:07:10.368 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Succeeded. 2023-07-10T17:07:10.368 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 systemd[1]: Stopped Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:07:10.368 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:09 smithi192 systemd[1]: Started Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:07:10.636 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:10 smithi192 systemd[1]: Stopping Ceph node-exporter.b for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:07:11.182 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:11 smithi192 bash[23730]: cluster 2023-07-10T17:07:09.980899+0000 mgr.x (mgr.24857) 50 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:11.182 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:11 smithi192 bash[23730]: audit 2023-07-10T17:07:10.029014+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:11.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:11 smithi116 bash[23740]: cluster 2023-07-10T17:07:09.980899+0000 mgr.x (mgr.24857) 50 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:11.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:11 smithi116 bash[23740]: audit 2023-07-10T17:07:10.029014+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:11.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:11 smithi116 bash[23740]: audit 2023-07-10T17:07:10.041107+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:11.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:11 smithi116 bash[23740]: cephadm 2023-07-10T17:07:10.084390+0000 mgr.x (mgr.24857) 51 : cephadm [INF] Reconfiguring node-exporter.b (dependencies changed)... 2023-07-10T17:07:11.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:11 smithi116 bash[23740]: cephadm 2023-07-10T17:07:10.085244+0000 mgr.x (mgr.24857) 52 : cephadm [INF] Reconfiguring daemon node-exporter.b on smithi192 2023-07-10T17:07:11.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:11 smithi116 bash[28351]: cluster 2023-07-10T17:07:09.980899+0000 mgr.x (mgr.24857) 50 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:11.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:11 smithi116 bash[28351]: audit 2023-07-10T17:07:10.029014+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:11.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:11 smithi116 bash[28351]: audit 2023-07-10T17:07:10.041107+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:11.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:11 smithi116 bash[28351]: cephadm 2023-07-10T17:07:10.084390+0000 mgr.x (mgr.24857) 51 : cephadm [INF] Reconfiguring node-exporter.b (dependencies changed)... 2023-07-10T17:07:11.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:11 smithi116 bash[28351]: cephadm 2023-07-10T17:07:10.085244+0000 mgr.x (mgr.24857) 52 : cephadm [INF] Reconfiguring daemon node-exporter.b on smithi192 2023-07-10T17:07:11.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:11 smithi192 bash[23730]: audit 2023-07-10T17:07:10.041107+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:11.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:11 smithi192 bash[23730]: cephadm 2023-07-10T17:07:10.084390+0000 mgr.x (mgr.24857) 51 : cephadm [INF] Reconfiguring node-exporter.b (dependencies changed)... 2023-07-10T17:07:11.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:11 smithi192 bash[23730]: cephadm 2023-07-10T17:07:10.085244+0000 mgr.x (mgr.24857) 52 : cephadm [INF] Reconfiguring daemon node-exporter.b on smithi192 2023-07-10T17:07:11.994 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:11 smithi192 docker[44423]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-node-exporter.b 2023-07-10T17:07:11.995 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:11 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2023-07-10T17:07:11.995 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:11 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@node-exporter.b.service: Failed with result 'exit-code'. 2023-07-10T17:07:11.995 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:11 smithi192 systemd[1]: Stopped Ceph node-exporter.b for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:07:11.995 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:11 smithi192 systemd[1]: Started Ceph node-exporter.b for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:07:12.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.091Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-10T17:07:12.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.091Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-10T17:07:12.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.091Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-10T17:07:12.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.091Z caller=main.go:339 host_details="(Linux 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 smithi192 (none))" 2023-07-10T17:07:12.388 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.091Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-10T17:07:12.388 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.091Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-10T17:07:12.388 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.093Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-10T17:07:12.388 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.093Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-10T17:07:12.388 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.100Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-10T17:07:12.388 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.100Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2023-07-10T17:07:12.388 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.114Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2023-07-10T17:07:12.389 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.115Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2023-07-10T17:07:12.389 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.115Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=14.592781ms 2023-07-10T17:07:12.389 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.119Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-10T17:07:12.389 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.119Z caller=main.go:695 msg="TSDB started" 2023-07-10T17:07:12.389 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.119Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-10T17:07:12.389 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.120Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-10T17:07:12.389 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.120Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-10T17:07:12.389 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.120Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-10T17:07:12.390 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.120Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-10T17:07:12.390 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.120Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-10T17:07:12.390 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.120Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-10T17:07:12.390 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.120Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-10T17:07:12.390 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.120Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-10T17:07:12.390 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.120Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-10T17:07:12.390 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=info ts=2023-07-10T17:07:12.120Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-10T17:07:12.391 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[44400]: level=error ts=2023-07-10T17:07:12.121Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-10T17:07:13.039 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[23740]: audit 2023-07-10T17:07:11.702811+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:13.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[23740]: audit 2023-07-10T17:07:11.713734+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:13.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[23740]: cephadm 2023-07-10T17:07:11.754186+0000 mgr.x (mgr.24857) 53 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2023-07-10T17:07:13.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[23740]: cephadm 2023-07-10T17:07:11.769017+0000 mgr.x (mgr.24857) 54 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2023-07-10T17:07:13.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[23740]: audit 2023-07-10T17:07:11.859292+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:13.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[23740]: audit 2023-07-10T17:07:11.868847+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:13.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[23740]: audit 2023-07-10T17:07:11.873735+0000 mon.b (mon.2) 60 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-10T17:07:13.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[23740]: audit 2023-07-10T17:07:11.874817+0000 mgr.x (mgr.24857) 55 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-10T17:07:13.041 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[23740]: cephadm 2023-07-10T17:07:11.878848+0000 mgr.x (mgr.24857) 56 : cephadm [INF] Reconfiguring daemon grafana.a on smithi192 2023-07-10T17:07:13.041 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[23740]: cluster 2023-07-10T17:07:11.982824+0000 mgr.x (mgr.24857) 57 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:13.041 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[23740]: audit 2023-07-10T17:07:12.659287+0000 mon.c (mon.1) 44 : audit [DBG] from='client.? 172.21.15.116:0/3285421854' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:07:13.041 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[28351]: audit 2023-07-10T17:07:11.702811+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:13.041 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[28351]: audit 2023-07-10T17:07:11.713734+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:13.042 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[28351]: cephadm 2023-07-10T17:07:11.754186+0000 mgr.x (mgr.24857) 53 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2023-07-10T17:07:13.042 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[28351]: cephadm 2023-07-10T17:07:11.769017+0000 mgr.x (mgr.24857) 54 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2023-07-10T17:07:13.042 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[28351]: audit 2023-07-10T17:07:11.859292+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:13.042 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[28351]: audit 2023-07-10T17:07:11.868847+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:13.042 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[28351]: audit 2023-07-10T17:07:11.873735+0000 mon.b (mon.2) 60 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-10T17:07:13.042 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[28351]: audit 2023-07-10T17:07:11.874817+0000 mgr.x (mgr.24857) 55 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-10T17:07:13.042 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[28351]: cephadm 2023-07-10T17:07:11.878848+0000 mgr.x (mgr.24857) 56 : cephadm [INF] Reconfiguring daemon grafana.a on smithi192 2023-07-10T17:07:13.042 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[28351]: cluster 2023-07-10T17:07:11.982824+0000 mgr.x (mgr.24857) 57 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:13.043 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:12 smithi116 bash[28351]: audit 2023-07-10T17:07:12.659287+0000 mon.c (mon.1) 44 : audit [DBG] from='client.? 172.21.15.116:0/3285421854' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:07:13.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[23730]: audit 2023-07-10T17:07:11.702811+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:13.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[23730]: audit 2023-07-10T17:07:11.713734+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:13.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[23730]: cephadm 2023-07-10T17:07:11.754186+0000 mgr.x (mgr.24857) 53 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2023-07-10T17:07:13.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[23730]: cephadm 2023-07-10T17:07:11.769017+0000 mgr.x (mgr.24857) 54 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2023-07-10T17:07:13.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[23730]: audit 2023-07-10T17:07:11.859292+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:13.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[23730]: audit 2023-07-10T17:07:11.868847+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:07:13.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[23730]: audit 2023-07-10T17:07:11.873735+0000 mon.b (mon.2) 60 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-10T17:07:13.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[23730]: audit 2023-07-10T17:07:11.874817+0000 mgr.x (mgr.24857) 55 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-10T17:07:13.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[23730]: cephadm 2023-07-10T17:07:11.878848+0000 mgr.x (mgr.24857) 56 : cephadm [INF] Reconfiguring daemon grafana.a on smithi192 2023-07-10T17:07:13.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[23730]: cluster 2023-07-10T17:07:11.982824+0000 mgr.x (mgr.24857) 57 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:13.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:12 smithi192 bash[23730]: audit 2023-07-10T17:07:12.659287+0000 mon.c (mon.1) 44 : audit [DBG] from='client.? 172.21.15.116:0/3285421854' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:07:13.726 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:13 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-10T17:07:13.727 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:13 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-10T17:07:14.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[23730]: audit 2023-07-10T17:07:13.029892+0000 mon.a (mon.0) 718 : audit [INF] from='client.? 172.21.15.116:0/1425357481' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: access denied 2023-07-10T17:07:14.137 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2023-07-10T17:07:14.137 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2023-07-10T17:07:14.138 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2023-07-10T17:07:14.138 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - arp" source="node_exporter.go:104" 2023-07-10T17:07:14.138 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - bcache" source="node_exporter.go:104" 2023-07-10T17:07:14.138 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - bonding" source="node_exporter.go:104" 2023-07-10T17:07:14.138 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2023-07-10T17:07:14.138 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - cpu" source="node_exporter.go:104" 2023-07-10T17:07:14.138 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2023-07-10T17:07:14.138 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2023-07-10T17:07:14.139 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - edac" source="node_exporter.go:104" 2023-07-10T17:07:14.139 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - entropy" source="node_exporter.go:104" 2023-07-10T17:07:14.139 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - filefd" source="node_exporter.go:104" 2023-07-10T17:07:14.139 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2023-07-10T17:07:14.139 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2023-07-10T17:07:14.139 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2023-07-10T17:07:14.139 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2023-07-10T17:07:14.140 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2023-07-10T17:07:14.140 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2023-07-10T17:07:14.140 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2023-07-10T17:07:14.140 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - netclass" source="node_exporter.go:104" 2023-07-10T17:07:14.140 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - netdev" source="node_exporter.go:104" 2023-07-10T17:07:14.140 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - netstat" source="node_exporter.go:104" 2023-07-10T17:07:14.140 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - nfs" source="node_exporter.go:104" 2023-07-10T17:07:14.140 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2023-07-10T17:07:14.141 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - pressure" source="node_exporter.go:104" 2023-07-10T17:07:14.141 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2023-07-10T17:07:14.141 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - stat" source="node_exporter.go:104" 2023-07-10T17:07:14.141 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - textfile" source="node_exporter.go:104" 2023-07-10T17:07:14.141 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - time" source="node_exporter.go:104" 2023-07-10T17:07:14.141 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - uname" source="node_exporter.go:104" 2023-07-10T17:07:14.141 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2023-07-10T17:07:14.141 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - xfs" source="node_exporter.go:104" 2023-07-10T17:07:14.142 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg=" - zfs" source="node_exporter.go:104" 2023-07-10T17:07:14.142 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:07:13 smithi192 bash[44474]: time="2023-07-10T17:07:13Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2023-07-10T17:07:14.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:13 smithi116 bash[23740]: audit 2023-07-10T17:07:13.029892+0000 mon.a (mon.0) 718 : audit [INF] from='client.? 172.21.15.116:0/1425357481' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: access denied 2023-07-10T17:07:14.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:13 smithi116 bash[28351]: audit 2023-07-10T17:07:13.029892+0000 mon.a (mon.0) 718 : audit [INF] from='client.? 172.21.15.116:0/1425357481' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: access denied 2023-07-10T17:07:15.019 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:07:15.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:14 smithi192 bash[23730]: cluster 2023-07-10T17:07:13.983604+0000 mgr.x (mgr.24857) 58 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:15.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:14 smithi116 bash[23740]: cluster 2023-07-10T17:07:13.983604+0000 mgr.x (mgr.24857) 58 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:15.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:14 smithi116 bash[28351]: cluster 2023-07-10T17:07:13.983604+0000 mgr.x (mgr.24857) 58 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:15.763 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:07:15.763 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 starting - - - - 2023-07-10T17:07:15.764 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (6m) 46s ago 6m 26.5M - 6.7.4 557c83e11646 ca7910fb45c5 2023-07-10T17:07:15.764 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 starting - - - - 2023-07-10T17:07:15.764 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443 running (66s) 46s ago 11m 460M - 18.1.2-30-g8507faa7 17e8f0d79247 d22a6b7f47df 2023-07-10T17:07:15.764 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:9283 running (12m) 46s ago 12m 354M - 16.2.0 24ecd6d5f14c 9776722e922e 2023-07-10T17:07:15.764 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (12m) 46s ago 12m 58.6M 2048M 16.2.0 24ecd6d5f14c 33f26d450a67 2023-07-10T17:07:15.765 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (11m) 46s ago 11m 52.3M 2048M 16.2.0 24ecd6d5f14c 9ad4329e5488 2023-07-10T17:07:15.765 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (11m) 46s ago 11m 48.4M 2048M 16.2.0 24ecd6d5f14c 31e3ac12741b 2023-07-10T17:07:15.765 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 starting - - - - 2023-07-10T17:07:15.765 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 starting - - - - 2023-07-10T17:07:15.765 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (10m) 46s ago 10m 52.8M 4096M 16.2.0 24ecd6d5f14c 4fa26367c830 2023-07-10T17:07:15.765 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (10m) 46s ago 10m 52.0M 4096M 16.2.0 24ecd6d5f14c c157509b3fd4 2023-07-10T17:07:15.765 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (9m) 46s ago 9m 48.1M 4096M 16.2.0 24ecd6d5f14c f1fd77b3c454 2023-07-10T17:07:15.766 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (9m) 46s ago 9m 53.4M 4096M 16.2.0 24ecd6d5f14c 60949d479c11 2023-07-10T17:07:15.766 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (8m) 46s ago 8m 50.9M 4096M 16.2.0 24ecd6d5f14c 75fc0b36aab8 2023-07-10T17:07:15.766 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (8m) 46s ago 8m 49.2M 4096M 16.2.0 24ecd6d5f14c d673065333dd 2023-07-10T17:07:15.766 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (8m) 46s ago 8m 49.4M 4096M 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:07:15.766 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (7m) 46s ago 7m 52.7M 4096M 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:07:15.766 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 starting - - - - 2023-07-10T17:07:15.767 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (4m) 46s ago 4m 52.8M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:07:15.767 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (4m) 46s ago 4m 52.6M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:07:15.767 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (4m) 46s ago 4m 53.0M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:07:15.767 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (4m) 46s ago 4m 54.6M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:07:16.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:15 smithi116 bash[28351]: audit 2023-07-10T17:07:14.994719+0000 mgr.x (mgr.24857) 59 : audit [DBG] from='client.25027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:16.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:15 smithi116 bash[28351]: audit 2023-07-10T17:07:15.373824+0000 mgr.x (mgr.24857) 60 : audit [DBG] from='client.25033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:16.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:15 smithi116 bash[23740]: audit 2023-07-10T17:07:14.994719+0000 mgr.x (mgr.24857) 59 : audit [DBG] from='client.25027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:16.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:15 smithi116 bash[23740]: audit 2023-07-10T17:07:15.373824+0000 mgr.x (mgr.24857) 60 : audit [DBG] from='client.25033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:16.199 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:07:16.199 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:07:16.199 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2023-07-10T17:07:16.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:07:16.199 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:07:16.200 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2023-07-10T17:07:16.200 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 1 2023-07-10T17:07:16.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:07:16.200 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:07:16.201 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-10T17:07:16.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:07:16.201 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": {}, 2023-07-10T17:07:16.201 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:07:16.202 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:07:16.202 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:07:16.202 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:07:16.203 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 16, 2023-07-10T17:07:16.203 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 1 2023-07-10T17:07:16.203 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:07:16.204 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:07:16.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:15 smithi192 bash[23730]: audit 2023-07-10T17:07:14.994719+0000 mgr.x (mgr.24857) 59 : audit [DBG] from='client.25027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:16.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:15 smithi192 bash[23730]: audit 2023-07-10T17:07:15.373824+0000 mgr.x (mgr.24857) 60 : audit [DBG] from='client.25033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:16.562 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:07:16.563 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:07:16.563 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:07:16.563 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:07:16.564 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [], 2023-07-10T17:07:16.564 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "1/23 daemons upgraded", 2023-07-10T17:07:16.564 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "", 2023-07-10T17:07:16.564 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:07:16.564 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:07:16.997 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2023-07-10T17:07:16.997 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2023-07-10T17:07:16.997 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:07:17.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:16 smithi192 bash[23730]: audit 2023-07-10T17:07:15.733624+0000 mgr.x (mgr.24857) 61 : audit [DBG] from='client.25039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:17.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:16 smithi192 bash[23730]: cluster 2023-07-10T17:07:15.985274+0000 mgr.x (mgr.24857) 62 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:17.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:16 smithi192 bash[23730]: audit 2023-07-10T17:07:16.195023+0000 mon.c (mon.1) 45 : audit [DBG] from='client.? 172.21.15.116:0/1162743049' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:07:17.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:16 smithi192 bash[23730]: audit 2023-07-10T17:07:16.558897+0000 mgr.x (mgr.24857) 63 : audit [DBG] from='client.25051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:17.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:16 smithi116 bash[23740]: audit 2023-07-10T17:07:15.733624+0000 mgr.x (mgr.24857) 61 : audit [DBG] from='client.25039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:17.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:16 smithi116 bash[23740]: cluster 2023-07-10T17:07:15.985274+0000 mgr.x (mgr.24857) 62 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:17.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:16 smithi116 bash[23740]: audit 2023-07-10T17:07:16.195023+0000 mon.c (mon.1) 45 : audit [DBG] from='client.? 172.21.15.116:0/1162743049' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:07:17.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:16 smithi116 bash[23740]: audit 2023-07-10T17:07:16.558897+0000 mgr.x (mgr.24857) 63 : audit [DBG] from='client.25051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:17.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:16 smithi116 bash[28351]: audit 2023-07-10T17:07:15.733624+0000 mgr.x (mgr.24857) 61 : audit [DBG] from='client.25039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:17.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:16 smithi116 bash[28351]: cluster 2023-07-10T17:07:15.985274+0000 mgr.x (mgr.24857) 62 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:17.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:16 smithi116 bash[28351]: audit 2023-07-10T17:07:16.195023+0000 mon.c (mon.1) 45 : audit [DBG] from='client.? 172.21.15.116:0/1162743049' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:07:17.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:16 smithi116 bash[28351]: audit 2023-07-10T17:07:16.558897+0000 mgr.x (mgr.24857) 63 : audit [DBG] from='client.25051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:18.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:17 smithi192 bash[23730]: audit 2023-07-10T17:07:16.993534+0000 mon.c (mon.1) 46 : audit [DBG] from='client.? 172.21.15.116:0/2788237992' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:07:18.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:17 smithi116 bash[23740]: audit 2023-07-10T17:07:16.993534+0000 mon.c (mon.1) 46 : audit [DBG] from='client.? 172.21.15.116:0/2788237992' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:07:18.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:17 smithi116 bash[28351]: audit 2023-07-10T17:07:16.993534+0000 mon.c (mon.1) 46 : audit [DBG] from='client.? 172.21.15.116:0/2788237992' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:07:19.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:19 smithi192 bash[23730]: cluster 2023-07-10T17:07:17.986201+0000 mgr.x (mgr.24857) 64 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:19.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:19 smithi116 bash[23740]: cluster 2023-07-10T17:07:17.986201+0000 mgr.x (mgr.24857) 64 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:19.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:19 smithi116 bash[28351]: cluster 2023-07-10T17:07:17.986201+0000 mgr.x (mgr.24857) 64 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:21.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:21 smithi192 bash[23730]: cluster 2023-07-10T17:07:19.987083+0000 mgr.x (mgr.24857) 65 : cluster [DBG] pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:21.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:21 smithi116 bash[23740]: cluster 2023-07-10T17:07:19.987083+0000 mgr.x (mgr.24857) 65 : cluster [DBG] pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:21.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:21 smithi116 bash[28351]: cluster 2023-07-10T17:07:19.987083+0000 mgr.x (mgr.24857) 65 : cluster [DBG] pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:23.858 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:23 smithi192 bash[23730]: cluster 2023-07-10T17:07:21.988922+0000 mgr.x (mgr.24857) 66 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:23.859 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:23 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 1. 2023-07-10T17:07:23.859 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:23 smithi192 systemd[1]: Stopped Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:07:23.859 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:23 smithi192 systemd[1]: Started Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:07:23.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:23 smithi116 bash[23740]: cluster 2023-07-10T17:07:21.988922+0000 mgr.x (mgr.24857) 66 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:23.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:23 smithi116 bash[28351]: cluster 2023-07-10T17:07:21.988922+0000 mgr.x (mgr.24857) 66 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:25.586 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.448Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-10T17:07:25.586 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.448Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-10T17:07:25.586 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.448Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-10T17:07:25.586 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.448Z caller=main.go:339 host_details="(Linux 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 smithi192 (none))" 2023-07-10T17:07:25.586 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.448Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-10T17:07:25.587 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.449Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-10T17:07:25.587 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.451Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-10T17:07:25.587 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.451Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-10T17:07:25.587 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.456Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-10T17:07:25.587 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.457Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2023-07-10T17:07:25.587 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.470Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2023-07-10T17:07:25.587 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.470Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2023-07-10T17:07:25.588 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.471Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2023-07-10T17:07:25.588 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.471Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=14.931606ms 2023-07-10T17:07:25.588 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.473Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-10T17:07:25.588 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.473Z caller=main.go:695 msg="TSDB started" 2023-07-10T17:07:25.588 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.473Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-10T17:07:25.588 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.473Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-10T17:07:25.588 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.473Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-10T17:07:25.588 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.474Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-10T17:07:25.589 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.474Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-10T17:07:25.589 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.474Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-10T17:07:25.589 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.474Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-10T17:07:25.589 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.474Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-10T17:07:25.589 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.474Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-10T17:07:25.589 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.474Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-10T17:07:25.589 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=info ts=2023-07-10T17:07:25.474Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-10T17:07:25.590 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[44657]: level=error ts=2023-07-10T17:07:25.474Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-10T17:07:25.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:25 smithi192 bash[23730]: cluster 2023-07-10T17:07:23.989715+0000 mgr.x (mgr.24857) 67 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:25.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:25 smithi116 bash[23740]: cluster 2023-07-10T17:07:23.989715+0000 mgr.x (mgr.24857) 67 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:25.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:25 smithi116 bash[28351]: cluster 2023-07-10T17:07:23.989715+0000 mgr.x (mgr.24857) 67 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:26.386 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:26 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-10T17:07:26.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:26 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-10T17:07:27.873 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:27 smithi116 bash[28351]: cluster 2023-07-10T17:07:25.991568+0000 mgr.x (mgr.24857) 68 : cluster [DBG] pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:27.874 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:27 smithi116 bash[28351]: audit 2023-07-10T17:07:27.285554+0000 mon.c (mon.1) 47 : audit [DBG] from='client.? 172.21.15.116:0/1111606034' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:07:27.874 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:27 smithi116 bash[23740]: cluster 2023-07-10T17:07:25.991568+0000 mgr.x (mgr.24857) 68 : cluster [DBG] pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:27.874 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:27 smithi116 bash[23740]: audit 2023-07-10T17:07:27.285554+0000 mon.c (mon.1) 47 : audit [DBG] from='client.? 172.21.15.116:0/1111606034' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:07:27.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:27 smithi192 bash[23730]: cluster 2023-07-10T17:07:25.991568+0000 mgr.x (mgr.24857) 68 : cluster [DBG] pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:27.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:27 smithi192 bash[23730]: audit 2023-07-10T17:07:27.285554+0000 mon.c (mon.1) 47 : audit [DBG] from='client.? 172.21.15.116:0/1111606034' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:07:28.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:28 smithi192 bash[23730]: audit 2023-07-10T17:07:27.656980+0000 mon.a (mon.0) 719 : audit [INF] from='client.? 172.21.15.116:0/2932742913' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: access denied 2023-07-10T17:07:28.922 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:28 smithi116 bash[23740]: audit 2023-07-10T17:07:27.656980+0000 mon.a (mon.0) 719 : audit [INF] from='client.? 172.21.15.116:0/2932742913' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: access denied 2023-07-10T17:07:28.923 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:28 smithi116 bash[28351]: audit 2023-07-10T17:07:27.656980+0000 mon.a (mon.0) 719 : audit [INF] from='client.? 172.21.15.116:0/2932742913' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: access denied 2023-07-10T17:07:29.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:29 smithi192 bash[23730]: cluster 2023-07-10T17:07:27.992406+0000 mgr.x (mgr.24857) 69 : cluster [DBG] pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:29.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:29 smithi116 bash[23740]: cluster 2023-07-10T17:07:27.992406+0000 mgr.x (mgr.24857) 69 : cluster [DBG] pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:29.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:29 smithi116 bash[28351]: cluster 2023-07-10T17:07:27.992406+0000 mgr.x (mgr.24857) 69 : cluster [DBG] pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:31.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:31 smithi192 bash[23730]: cluster 2023-07-10T17:07:29.993173+0000 mgr.x (mgr.24857) 70 : cluster [DBG] pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:31.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:31 smithi116 bash[23740]: cluster 2023-07-10T17:07:29.993173+0000 mgr.x (mgr.24857) 70 : cluster [DBG] pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:31.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:31 smithi116 bash[28351]: cluster 2023-07-10T17:07:29.993173+0000 mgr.x (mgr.24857) 70 : cluster [DBG] pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:33.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:33 smithi192 bash[23730]: cluster 2023-07-10T17:07:31.994931+0000 mgr.x (mgr.24857) 71 : cluster [DBG] pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:33.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:33 smithi116 bash[28351]: cluster 2023-07-10T17:07:31.994931+0000 mgr.x (mgr.24857) 71 : cluster [DBG] pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:33.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:33 smithi116 bash[23740]: cluster 2023-07-10T17:07:31.994931+0000 mgr.x (mgr.24857) 71 : cluster [DBG] pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:35.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:35 smithi192 bash[23730]: cluster 2023-07-10T17:07:33.995696+0000 mgr.x (mgr.24857) 72 : cluster [DBG] pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:35.892 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:35 smithi116 bash[28351]: cluster 2023-07-10T17:07:33.995696+0000 mgr.x (mgr.24857) 72 : cluster [DBG] pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:35.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:35 smithi116 bash[23740]: cluster 2023-07-10T17:07:33.995696+0000 mgr.x (mgr.24857) 72 : cluster [DBG] pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:36.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:36 smithi192 bash[23730]: cluster 2023-07-10T17:07:35.997369+0000 mgr.x (mgr.24857) 73 : cluster [DBG] pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:36.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:36 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 2. 2023-07-10T17:07:36.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:36 smithi192 systemd[1]: Stopped Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:07:36.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:36 smithi192 systemd[1]: Started Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:07:36.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:36 smithi116 bash[23740]: cluster 2023-07-10T17:07:35.997369+0000 mgr.x (mgr.24857) 73 : cluster [DBG] pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:36.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:36 smithi116 bash[28351]: cluster 2023-07-10T17:07:35.997369+0000 mgr.x (mgr.24857) 73 : cluster [DBG] pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:38.285 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.085Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-10T17:07:38.286 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.085Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-10T17:07:38.286 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.085Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-10T17:07:38.286 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.085Z caller=main.go:339 host_details="(Linux 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 smithi192 (none))" 2023-07-10T17:07:38.286 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.085Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-10T17:07:38.286 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.086Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-10T17:07:38.286 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.087Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-10T17:07:38.286 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.087Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-10T17:07:38.287 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.092Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-10T17:07:38.287 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.093Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2023-07-10T17:07:38.287 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.105Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2023-07-10T17:07:38.287 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.105Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2023-07-10T17:07:38.287 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.106Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2023-07-10T17:07:38.287 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.106Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2023-07-10T17:07:38.287 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.106Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=13.814854ms 2023-07-10T17:07:38.288 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.109Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-10T17:07:38.288 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.109Z caller=main.go:695 msg="TSDB started" 2023-07-10T17:07:38.288 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.109Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-10T17:07:38.288 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.110Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-10T17:07:38.288 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.110Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-10T17:07:38.288 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.110Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-10T17:07:38.288 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.110Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-10T17:07:38.289 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.110Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-10T17:07:38.289 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.110Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-10T17:07:38.289 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.110Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-10T17:07:38.289 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.110Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-10T17:07:38.289 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.110Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-10T17:07:38.289 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=info ts=2023-07-10T17:07:38.110Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-10T17:07:38.289 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 bash[44768]: level=error ts=2023-07-10T17:07:38.110Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-10T17:07:39.136 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-10T17:07:39.136 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:38 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-10T17:07:39.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:39 smithi116 bash[23740]: cluster 2023-07-10T17:07:37.998297+0000 mgr.x (mgr.24857) 74 : cluster [DBG] pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:39.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:39 smithi192 bash[23730]: cluster 2023-07-10T17:07:37.998297+0000 mgr.x (mgr.24857) 74 : cluster [DBG] pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:39.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:39 smithi116 bash[28351]: cluster 2023-07-10T17:07:37.998297+0000 mgr.x (mgr.24857) 74 : cluster [DBG] pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:41.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:41 smithi192 bash[23730]: cluster 2023-07-10T17:07:39.999073+0000 mgr.x (mgr.24857) 75 : cluster [DBG] pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:41.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:41 smithi116 bash[23740]: cluster 2023-07-10T17:07:39.999073+0000 mgr.x (mgr.24857) 75 : cluster [DBG] pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:41.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:41 smithi116 bash[28351]: cluster 2023-07-10T17:07:39.999073+0000 mgr.x (mgr.24857) 75 : cluster [DBG] pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:42.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:42 smithi116 bash[28351]: audit 2023-07-10T17:07:41.776490+0000 mon.a (mon.0) 720 : audit [DBG] from='client.? 172.21.15.116:0/2614545993' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:07:42.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:42 smithi116 bash[28351]: audit 2023-07-10T17:07:42.149464+0000 mon.a (mon.0) 721 : audit [INF] from='client.? 172.21.15.116:0/2369428154' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: access denied 2023-07-10T17:07:42.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:42 smithi116 bash[23740]: audit 2023-07-10T17:07:41.776490+0000 mon.a (mon.0) 720 : audit [DBG] from='client.? 172.21.15.116:0/2614545993' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:07:42.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:42 smithi116 bash[23740]: audit 2023-07-10T17:07:42.149464+0000 mon.a (mon.0) 721 : audit [INF] from='client.? 172.21.15.116:0/2369428154' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: access denied 2023-07-10T17:07:42.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:42 smithi192 bash[23730]: audit 2023-07-10T17:07:41.776490+0000 mon.a (mon.0) 720 : audit [DBG] from='client.? 172.21.15.116:0/2614545993' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:07:42.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:42 smithi192 bash[23730]: audit 2023-07-10T17:07:42.149464+0000 mon.a (mon.0) 721 : audit [INF] from='client.? 172.21.15.116:0/2369428154' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: access denied 2023-07-10T17:07:43.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:43 smithi192 bash[23730]: cluster 2023-07-10T17:07:42.000798+0000 mgr.x (mgr.24857) 76 : cluster [DBG] pgmap v47: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:43.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:43 smithi116 bash[23740]: cluster 2023-07-10T17:07:42.000798+0000 mgr.x (mgr.24857) 76 : cluster [DBG] pgmap v47: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:43.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:43 smithi116 bash[28351]: cluster 2023-07-10T17:07:42.000798+0000 mgr.x (mgr.24857) 76 : cluster [DBG] pgmap v47: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:45.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:45 smithi192 bash[23730]: cluster 2023-07-10T17:07:44.001572+0000 mgr.x (mgr.24857) 77 : cluster [DBG] pgmap v48: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:45.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:45 smithi116 bash[23740]: cluster 2023-07-10T17:07:44.001572+0000 mgr.x (mgr.24857) 77 : cluster [DBG] pgmap v48: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:45.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:45 smithi116 bash[28351]: cluster 2023-07-10T17:07:44.001572+0000 mgr.x (mgr.24857) 77 : cluster [DBG] pgmap v48: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:47.395 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:07:47.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:47 smithi192 bash[23730]: cluster 2023-07-10T17:07:46.003244+0000 mgr.x (mgr.24857) 78 : cluster [DBG] pgmap v49: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:47.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:47 smithi116 bash[23740]: cluster 2023-07-10T17:07:46.003244+0000 mgr.x (mgr.24857) 78 : cluster [DBG] pgmap v49: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:47.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:47 smithi116 bash[28351]: cluster 2023-07-10T17:07:46.003244+0000 mgr.x (mgr.24857) 78 : cluster [DBG] pgmap v49: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:48.132 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:07:48.132 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 starting - - - - 2023-07-10T17:07:48.132 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (6m) 78s ago 6m 26.5M - 6.7.4 557c83e11646 ca7910fb45c5 2023-07-10T17:07:48.133 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 starting - - - - 2023-07-10T17:07:48.133 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443 running (98s) 78s ago 11m 460M - 18.1.2-30-g8507faa7 17e8f0d79247 d22a6b7f47df 2023-07-10T17:07:48.133 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:9283 running (13m) 79s ago 13m 354M - 16.2.0 24ecd6d5f14c 9776722e922e 2023-07-10T17:07:48.133 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (13m) 79s ago 13m 58.6M 2048M 16.2.0 24ecd6d5f14c 33f26d450a67 2023-07-10T17:07:48.133 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (11m) 78s ago 11m 52.3M 2048M 16.2.0 24ecd6d5f14c 9ad4329e5488 2023-07-10T17:07:48.133 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (11m) 79s ago 11m 48.4M 2048M 16.2.0 24ecd6d5f14c 31e3ac12741b 2023-07-10T17:07:48.133 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 starting - - - - 2023-07-10T17:07:48.134 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 starting - - - - 2023-07-10T17:07:48.134 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (11m) 79s ago 11m 52.8M 4096M 16.2.0 24ecd6d5f14c 4fa26367c830 2023-07-10T17:07:48.134 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (10m) 79s ago 10m 52.0M 4096M 16.2.0 24ecd6d5f14c c157509b3fd4 2023-07-10T17:07:48.134 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (10m) 79s ago 10m 48.1M 4096M 16.2.0 24ecd6d5f14c f1fd77b3c454 2023-07-10T17:07:48.134 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (9m) 79s ago 9m 53.4M 4096M 16.2.0 24ecd6d5f14c 60949d479c11 2023-07-10T17:07:48.134 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (9m) 78s ago 9m 50.9M 4096M 16.2.0 24ecd6d5f14c 75fc0b36aab8 2023-07-10T17:07:48.134 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (9m) 78s ago 9m 49.2M 4096M 16.2.0 24ecd6d5f14c d673065333dd 2023-07-10T17:07:48.135 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (8m) 78s ago 8m 49.4M 4096M 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:07:48.135 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (8m) 78s ago 8m 52.7M 4096M 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:07:48.135 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 starting - - - - 2023-07-10T17:07:48.135 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (5m) 79s ago 5m 52.8M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:07:48.135 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (5m) 78s ago 5m 52.6M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:07:48.135 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (5m) 79s ago 5m 53.0M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:07:48.135 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (4m) 78s ago 4m 54.6M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:07:48.551 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:07:48.551 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:07:48.552 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2023-07-10T17:07:48.553 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:07:48.553 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:07:48.553 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2023-07-10T17:07:48.553 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 1 2023-07-10T17:07:48.553 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:07:48.553 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:07:48.553 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-10T17:07:48.554 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:07:48.554 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": {}, 2023-07-10T17:07:48.554 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:07:48.554 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:07:48.554 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:07:48.554 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:07:48.554 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 16, 2023-07-10T17:07:48.554 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 1 2023-07-10T17:07:48.555 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:07:48.555 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:07:48.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:48 smithi192 bash[23730]: audit 2023-07-10T17:07:47.377368+0000 mgr.x (mgr.24857) 79 : audit [DBG] from='client.25111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:48.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:48 smithi192 bash[23730]: audit 2023-07-10T17:07:48.547549+0000 mon.c (mon.1) 48 : audit [DBG] from='client.? 172.21.15.116:0/1464538081' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:07:48.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:48 smithi116 bash[28351]: audit 2023-07-10T17:07:47.377368+0000 mgr.x (mgr.24857) 79 : audit [DBG] from='client.25111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:48.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:48 smithi116 bash[28351]: audit 2023-07-10T17:07:48.547549+0000 mon.c (mon.1) 48 : audit [DBG] from='client.? 172.21.15.116:0/1464538081' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:07:48.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:48 smithi116 bash[23740]: audit 2023-07-10T17:07:47.377368+0000 mgr.x (mgr.24857) 79 : audit [DBG] from='client.25111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:48.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:48 smithi116 bash[23740]: audit 2023-07-10T17:07:48.547549+0000 mon.c (mon.1) 48 : audit [DBG] from='client.? 172.21.15.116:0/1464538081' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:07:48.920 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:07:48.921 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:07:48.921 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:07:48.921 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:07:48.921 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [], 2023-07-10T17:07:48.921 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "1/23 daemons upgraded", 2023-07-10T17:07:48.921 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "", 2023-07-10T17:07:48.922 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:07:48.922 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:07:49.354 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2023-07-10T17:07:49.354 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2023-07-10T17:07:49.355 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:07:49.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:48 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 3. 2023-07-10T17:07:49.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:48 smithi192 systemd[1]: Stopped Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:07:49.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:48 smithi192 systemd[1]: Started Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:07:49.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:49 smithi192 bash[23730]: audit 2023-07-10T17:07:47.752144+0000 mgr.x (mgr.24857) 80 : audit [DBG] from='client.25117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:49.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:49 smithi192 bash[23730]: cluster 2023-07-10T17:07:48.003990+0000 mgr.x (mgr.24857) 81 : cluster [DBG] pgmap v50: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:49.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:49 smithi192 bash[23730]: audit 2023-07-10T17:07:48.107026+0000 mgr.x (mgr.24857) 82 : audit [DBG] from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:49.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:49 smithi192 bash[23730]: audit 2023-07-10T17:07:49.351129+0000 mon.a (mon.0) 722 : audit [DBG] from='client.? 172.21.15.116:0/3891284898' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:07:49.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:49 smithi116 bash[23740]: audit 2023-07-10T17:07:47.752144+0000 mgr.x (mgr.24857) 80 : audit [DBG] from='client.25117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:49.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:49 smithi116 bash[23740]: cluster 2023-07-10T17:07:48.003990+0000 mgr.x (mgr.24857) 81 : cluster [DBG] pgmap v50: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:49.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:49 smithi116 bash[23740]: audit 2023-07-10T17:07:48.107026+0000 mgr.x (mgr.24857) 82 : audit [DBG] from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:49.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:49 smithi116 bash[23740]: audit 2023-07-10T17:07:49.351129+0000 mon.a (mon.0) 722 : audit [DBG] from='client.? 172.21.15.116:0/3891284898' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:07:49.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:49 smithi116 bash[28351]: audit 2023-07-10T17:07:47.752144+0000 mgr.x (mgr.24857) 80 : audit [DBG] from='client.25117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:49.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:49 smithi116 bash[28351]: cluster 2023-07-10T17:07:48.003990+0000 mgr.x (mgr.24857) 81 : cluster [DBG] pgmap v50: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:49.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:49 smithi116 bash[28351]: audit 2023-07-10T17:07:48.107026+0000 mgr.x (mgr.24857) 82 : audit [DBG] from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:49.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:49 smithi116 bash[28351]: audit 2023-07-10T17:07:49.351129+0000 mon.a (mon.0) 722 : audit [DBG] from='client.? 172.21.15.116:0/3891284898' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:07:50.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[23730]: audit 2023-07-10T17:07:48.916876+0000 mgr.x (mgr.24857) 83 : audit [DBG] from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:50.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.777Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-10T17:07:50.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.777Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-10T17:07:50.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.777Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-10T17:07:50.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.777Z caller=main.go:339 host_details="(Linux 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 smithi192 (none))" 2023-07-10T17:07:50.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.777Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-10T17:07:50.888 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.777Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-10T17:07:50.888 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.779Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-10T17:07:50.888 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.779Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-10T17:07:50.888 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.785Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-10T17:07:50.888 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.786Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=5 2023-07-10T17:07:50.888 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.801Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=5 2023-07-10T17:07:50.888 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.802Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=5 2023-07-10T17:07:50.889 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.803Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=5 2023-07-10T17:07:50.889 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.806Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=5 2023-07-10T17:07:50.889 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.806Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=5 2023-07-10T17:07:50.889 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.806Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=20.933958ms 2023-07-10T17:07:50.889 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.809Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-10T17:07:50.889 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.809Z caller=main.go:695 msg="TSDB started" 2023-07-10T17:07:50.889 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.809Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-10T17:07:50.890 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.809Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-10T17:07:50.890 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.809Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-10T17:07:50.890 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.809Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-10T17:07:50.890 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.809Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-10T17:07:50.890 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.809Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-10T17:07:50.890 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.809Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-10T17:07:50.890 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.809Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-10T17:07:50.891 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.809Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-10T17:07:50.891 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.809Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-10T17:07:50.891 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=info ts=2023-07-10T17:07:50.809Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-10T17:07:50.891 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:50 smithi192 bash[44882]: level=error ts=2023-07-10T17:07:50.810Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-10T17:07:50.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:50 smithi116 bash[23740]: audit 2023-07-10T17:07:48.916876+0000 mgr.x (mgr.24857) 83 : audit [DBG] from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:50.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:50 smithi116 bash[28351]: audit 2023-07-10T17:07:48.916876+0000 mgr.x (mgr.24857) 83 : audit [DBG] from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:07:51.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:51 smithi192 bash[23730]: cluster 2023-07-10T17:07:50.004792+0000 mgr.x (mgr.24857) 84 : cluster [DBG] pgmap v51: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:51.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:51 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-10T17:07:51.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:07:51 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-10T17:07:51.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:51 smithi116 bash[23740]: cluster 2023-07-10T17:07:50.004792+0000 mgr.x (mgr.24857) 84 : cluster [DBG] pgmap v51: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:51.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:51 smithi116 bash[28351]: cluster 2023-07-10T17:07:50.004792+0000 mgr.x (mgr.24857) 84 : cluster [DBG] pgmap v51: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:52.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:52 smithi192 bash[23730]: cluster 2023-07-10T17:07:52.006287+0000 mgr.x (mgr.24857) 85 : cluster [DBG] pgmap v52: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:52.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:52 smithi116 bash[23740]: cluster 2023-07-10T17:07:52.006287+0000 mgr.x (mgr.24857) 85 : cluster [DBG] pgmap v52: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:52.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:52 smithi116 bash[28351]: cluster 2023-07-10T17:07:52.006287+0000 mgr.x (mgr.24857) 85 : cluster [DBG] pgmap v52: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:55.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:55 smithi116 bash[28351]: cluster 2023-07-10T17:07:54.006920+0000 mgr.x (mgr.24857) 86 : cluster [DBG] pgmap v53: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:55.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:55 smithi116 bash[23740]: cluster 2023-07-10T17:07:54.006920+0000 mgr.x (mgr.24857) 86 : cluster [DBG] pgmap v53: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:55.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:55 smithi192 bash[23730]: cluster 2023-07-10T17:07:54.006920+0000 mgr.x (mgr.24857) 86 : cluster [DBG] pgmap v53: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:56.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:56 smithi192 bash[23730]: audit 2023-07-10T17:07:56.071924+0000 mon.c (mon.1) 49 : audit [DBG] from='client.? 172.21.15.116:0/4044857192' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:07:56.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:56 smithi192 bash[23730]: audit 2023-07-10T17:07:56.439760+0000 mon.c (mon.1) 50 : audit [INF] from='client.? 172.21.15.116:0/470023289' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: access denied 2023-07-10T17:07:56.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:56 smithi116 bash[23740]: audit 2023-07-10T17:07:56.071924+0000 mon.c (mon.1) 49 : audit [DBG] from='client.? 172.21.15.116:0/4044857192' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:07:56.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:56 smithi116 bash[23740]: audit 2023-07-10T17:07:56.439760+0000 mon.c (mon.1) 50 : audit [INF] from='client.? 172.21.15.116:0/470023289' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: access denied 2023-07-10T17:07:56.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:56 smithi116 bash[28351]: audit 2023-07-10T17:07:56.071924+0000 mon.c (mon.1) 49 : audit [DBG] from='client.? 172.21.15.116:0/4044857192' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:07:56.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:56 smithi116 bash[28351]: audit 2023-07-10T17:07:56.439760+0000 mon.c (mon.1) 50 : audit [INF] from='client.? 172.21.15.116:0/470023289' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: access denied 2023-07-10T17:07:57.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:57 smithi192 bash[23730]: cluster 2023-07-10T17:07:56.008134+0000 mgr.x (mgr.24857) 87 : cluster [DBG] pgmap v54: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:57.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:57 smithi116 bash[23740]: cluster 2023-07-10T17:07:56.008134+0000 mgr.x (mgr.24857) 87 : cluster [DBG] pgmap v54: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:57.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:57 smithi116 bash[28351]: cluster 2023-07-10T17:07:56.008134+0000 mgr.x (mgr.24857) 87 : cluster [DBG] pgmap v54: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:07:59.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:07:59 smithi192 bash[23730]: cluster 2023-07-10T17:07:58.008939+0000 mgr.x (mgr.24857) 88 : cluster [DBG] pgmap v55: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:59.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:07:59 smithi116 bash[23740]: cluster 2023-07-10T17:07:58.008939+0000 mgr.x (mgr.24857) 88 : cluster [DBG] pgmap v55: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:07:59.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:07:59 smithi116 bash[28351]: cluster 2023-07-10T17:07:58.008939+0000 mgr.x (mgr.24857) 88 : cluster [DBG] pgmap v55: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:01.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:01 smithi192 bash[23730]: cluster 2023-07-10T17:08:00.009754+0000 mgr.x (mgr.24857) 89 : cluster [DBG] pgmap v56: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:01.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:01 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 4. 2023-07-10T17:08:01.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:01 smithi192 systemd[1]: Stopped Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:08:01.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:01 smithi192 systemd[1]: Started Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:08:01.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:01 smithi116 bash[28351]: cluster 2023-07-10T17:08:00.009754+0000 mgr.x (mgr.24857) 89 : cluster [DBG] pgmap v56: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:01.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:01 smithi116 bash[23740]: cluster 2023-07-10T17:08:00.009754+0000 mgr.x (mgr.24857) 89 : cluster [DBG] pgmap v56: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:03.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:03 smithi192 bash[23730]: cluster 2023-07-10T17:08:02.011397+0000 mgr.x (mgr.24857) 90 : cluster [DBG] pgmap v57: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:03.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:03 smithi116 bash[23740]: cluster 2023-07-10T17:08:02.011397+0000 mgr.x (mgr.24857) 90 : cluster [DBG] pgmap v57: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:03.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:03 smithi116 bash[28351]: cluster 2023-07-10T17:08:02.011397+0000 mgr.x (mgr.24857) 90 : cluster [DBG] pgmap v57: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:05.350 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.088Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-10T17:08:05.351 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.088Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-10T17:08:05.351 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.088Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-10T17:08:05.351 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.088Z caller=main.go:339 host_details="(Linux 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 smithi192 (none))" 2023-07-10T17:08:05.351 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.088Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-10T17:08:05.351 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.088Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-10T17:08:05.351 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.090Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-10T17:08:05.351 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.090Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-10T17:08:05.352 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.093Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-10T17:08:05.352 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.094Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=6 2023-07-10T17:08:05.352 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.110Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=6 2023-07-10T17:08:05.352 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.110Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=6 2023-07-10T17:08:05.352 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.110Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=6 2023-07-10T17:08:05.352 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.110Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=6 2023-07-10T17:08:05.352 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.111Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=6 2023-07-10T17:08:05.352 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.111Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=6 2023-07-10T17:08:05.353 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.111Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=17.599051ms 2023-07-10T17:08:05.353 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.114Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-10T17:08:05.353 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.114Z caller=main.go:695 msg="TSDB started" 2023-07-10T17:08:05.353 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.114Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-10T17:08:05.353 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.115Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-10T17:08:05.353 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.115Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-10T17:08:05.353 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.115Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-10T17:08:05.354 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.115Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-10T17:08:05.354 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.115Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-10T17:08:05.354 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.115Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-10T17:08:05.354 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.115Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-10T17:08:05.354 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.115Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-10T17:08:05.354 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.115Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-10T17:08:05.354 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=info ts=2023-07-10T17:08:05.115Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-10T17:08:05.354 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[45020]: level=error ts=2023-07-10T17:08:05.116Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-10T17:08:05.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:05 smithi192 bash[23730]: cluster 2023-07-10T17:08:04.012138+0000 mgr.x (mgr.24857) 91 : cluster [DBG] pgmap v58: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:05.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:05 smithi116 bash[23740]: cluster 2023-07-10T17:08:04.012138+0000 mgr.x (mgr.24857) 91 : cluster [DBG] pgmap v58: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:05.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:05 smithi116 bash[28351]: cluster 2023-07-10T17:08:04.012138+0000 mgr.x (mgr.24857) 91 : cluster [DBG] pgmap v58: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:06.636 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:06 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-10T17:08:06.637 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:06 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-10T17:08:07.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:07 smithi192 bash[23730]: cluster 2023-07-10T17:08:06.013816+0000 mgr.x (mgr.24857) 92 : cluster [DBG] pgmap v59: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:07.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:07 smithi116 bash[23740]: cluster 2023-07-10T17:08:06.013816+0000 mgr.x (mgr.24857) 92 : cluster [DBG] pgmap v59: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:07.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:07 smithi116 bash[28351]: cluster 2023-07-10T17:08:06.013816+0000 mgr.x (mgr.24857) 92 : cluster [DBG] pgmap v59: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:08.272 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:07 smithi192 systemd[1]: Stopping Ceph grafana.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:08:08.272 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:08 smithi192 bash[37887]: t=2023-07-10T17:08:08+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2023-07-10T17:08:08.272 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:08 smithi192 bash[37887]: t=2023-07-10T17:08:08+0000 lvl=info msg="Stopped RemoteCache" logger=server reason="context canceled" 2023-07-10T17:08:08.272 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:08 smithi192 bash[37887]: t=2023-07-10T17:08:08+0000 lvl=info msg="Stopped UserAuthTokenService" logger=server reason="context canceled" 2023-07-10T17:08:08.272 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:08 smithi192 bash[37887]: t=2023-07-10T17:08:08+0000 lvl=info msg="Stopped InternalMetricsService" logger=server reason="context canceled" 2023-07-10T17:08:08.273 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:08 smithi192 bash[37887]: t=2023-07-10T17:08:08+0000 lvl=info msg="Stopped UsageStatsService" logger=server reason="context canceled" 2023-07-10T17:08:08.273 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:08 smithi192 bash[37887]: t=2023-07-10T17:08:08+0000 lvl=info msg="Stopped BackendPluginManager" logger=server reason="context canceled" 2023-07-10T17:08:08.273 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:08 smithi192 bash[37887]: t=2023-07-10T17:08:08+0000 lvl=info msg="Stopped NotificationService" logger=server reason="context canceled" 2023-07-10T17:08:08.273 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:08 smithi192 bash[37887]: t=2023-07-10T17:08:08+0000 lvl=info msg="Stopped provisioningServiceImpl" logger=server reason="context canceled" 2023-07-10T17:08:08.273 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:08 smithi192 bash[37887]: t=2023-07-10T17:08:08+0000 lvl=info msg="Stopped CleanUpService" logger=server reason="context canceled" 2023-07-10T17:08:08.273 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:08 smithi192 bash[37887]: t=2023-07-10T17:08:08+0000 lvl=info msg="Stopped Stream Manager" 2023-07-10T17:08:08.273 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:08 smithi192 bash[37887]: t=2023-07-10T17:08:08+0000 lvl=info msg="Stopped AlertEngine" logger=server reason="context canceled" 2023-07-10T17:08:08.274 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:08 smithi192 bash[37887]: t=2023-07-10T17:08:08+0000 lvl=info msg="Stopped PluginManager" logger=server reason="context canceled" 2023-07-10T17:08:09.138 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:08 smithi192 docker[45298]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-grafana.a 2023-07-10T17:08:09.138 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:08 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@grafana.a.service: Succeeded. 2023-07-10T17:08:09.138 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:08 smithi192 systemd[1]: Stopped Ceph grafana.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:08:09.138 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:08 smithi192 systemd[1]: Started Ceph grafana.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:08:09.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:09 smithi192 bash[23730]: cluster 2023-07-10T17:08:08.014570+0000 mgr.x (mgr.24857) 93 : cluster [DBG] pgmap v60: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:09.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:09 smithi192 bash[23730]: audit 2023-07-10T17:08:08.862181+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:09 smithi192 bash[23730]: audit 2023-07-10T17:08:08.873951+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:09 smithi192 bash[23730]: audit 2023-07-10T17:08:08.883283+0000 mon.b (mon.2) 61 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:08:09.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:09 smithi192 bash[23730]: audit 2023-07-10T17:08:08.885102+0000 mon.b (mon.2) 62 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi116.front.sepia.ceph.com:9093"}]: dispatch 2023-07-10T17:08:09.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:09 smithi192 bash[23730]: audit 2023-07-10T17:08:08.893133+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:09 smithi192 bash[23730]: audit 2023-07-10T17:08:08.904959+0000 mon.b (mon.2) 63 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:08:09.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:09 smithi192 bash[23730]: audit 2023-07-10T17:08:08.925422+0000 mon.b (mon.2) 64 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:08:09.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:09 smithi192 bash[23730]: audit 2023-07-10T17:08:08.935809+0000 mon.b (mon.2) 65 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:08:09.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:09 smithi192 bash[23730]: audit 2023-07-10T17:08:08.938497+0000 mon.b (mon.2) 66 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2023-07-10T17:08:09.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:09 smithi192 bash[23730]: audit 2023-07-10T17:08:08.948381+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:09 smithi192 bash[23730]: audit 2023-07-10T17:08:08.972470+0000 mon.b (mon.2) 67 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:08:09.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:09 smithi192 bash[23730]: audit 2023-07-10T17:08:08.977084+0000 mon.b (mon.2) 68 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2023-07-10T17:08:09.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:09 smithi192 bash[23730]: audit 2023-07-10T17:08:08.987078+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:09 smithi192 bash[23730]: audit 2023-07-10T17:08:09.019002+0000 mon.b (mon.2) 69 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:08:09.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:09 smithi192 bash[23730]: audit 2023-07-10T17:08:09.029121+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[23740]: cluster 2023-07-10T17:08:08.014570+0000 mgr.x (mgr.24857) 93 : cluster [DBG] pgmap v60: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:09.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[23740]: audit 2023-07-10T17:08:08.862181+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[23740]: audit 2023-07-10T17:08:08.873951+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[23740]: audit 2023-07-10T17:08:08.883283+0000 mon.b (mon.2) 61 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:08:09.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[23740]: audit 2023-07-10T17:08:08.885102+0000 mon.b (mon.2) 62 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi116.front.sepia.ceph.com:9093"}]: dispatch 2023-07-10T17:08:09.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[23740]: audit 2023-07-10T17:08:08.893133+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[23740]: audit 2023-07-10T17:08:08.904959+0000 mon.b (mon.2) 63 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:08:09.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[23740]: audit 2023-07-10T17:08:08.925422+0000 mon.b (mon.2) 64 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:08:09.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[23740]: audit 2023-07-10T17:08:08.935809+0000 mon.b (mon.2) 65 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:08:09.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[23740]: audit 2023-07-10T17:08:08.938497+0000 mon.b (mon.2) 66 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2023-07-10T17:08:09.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[23740]: audit 2023-07-10T17:08:08.948381+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[23740]: audit 2023-07-10T17:08:08.972470+0000 mon.b (mon.2) 67 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:08:09.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[23740]: audit 2023-07-10T17:08:08.977084+0000 mon.b (mon.2) 68 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2023-07-10T17:08:09.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[23740]: audit 2023-07-10T17:08:08.987078+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[23740]: audit 2023-07-10T17:08:09.019002+0000 mon.b (mon.2) 69 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:08:09.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[23740]: audit 2023-07-10T17:08:09.029121+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[28351]: cluster 2023-07-10T17:08:08.014570+0000 mgr.x (mgr.24857) 93 : cluster [DBG] pgmap v60: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:09.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[28351]: audit 2023-07-10T17:08:08.862181+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[28351]: audit 2023-07-10T17:08:08.873951+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[28351]: audit 2023-07-10T17:08:08.883283+0000 mon.b (mon.2) 61 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:08:09.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[28351]: audit 2023-07-10T17:08:08.885102+0000 mon.b (mon.2) 62 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi116.front.sepia.ceph.com:9093"}]: dispatch 2023-07-10T17:08:09.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[28351]: audit 2023-07-10T17:08:08.893133+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[28351]: audit 2023-07-10T17:08:08.904959+0000 mon.b (mon.2) 63 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:08:09.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[28351]: audit 2023-07-10T17:08:08.925422+0000 mon.b (mon.2) 64 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:08:09.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[28351]: audit 2023-07-10T17:08:08.935809+0000 mon.b (mon.2) 65 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:08:09.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[28351]: audit 2023-07-10T17:08:08.938497+0000 mon.b (mon.2) 66 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2023-07-10T17:08:09.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[28351]: audit 2023-07-10T17:08:08.948381+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[28351]: audit 2023-07-10T17:08:08.972470+0000 mon.b (mon.2) 67 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:08:09.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[28351]: audit 2023-07-10T17:08:08.977084+0000 mon.b (mon.2) 68 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2023-07-10T17:08:09.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[28351]: audit 2023-07-10T17:08:08.987078+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:09.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[28351]: audit 2023-07-10T17:08:09.019002+0000 mon.b (mon.2) 69 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:08:09.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:09 smithi116 bash[28351]: audit 2023-07-10T17:08:09.029121+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:10.196 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Starting Grafana" logger=server version=6.7.4 commit=8e44bbc5f5 branch=HEAD compiled=2020-05-26T17:35:38+0000 2023-07-10T17:08:10.196 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2023-07-10T17:08:10.196 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2023-07-10T17:08:10.196 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2023-07-10T17:08:10.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2023-07-10T17:08:10.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2023-07-10T17:08:10.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2023-07-10T17:08:10.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2023-07-10T17:08:10.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2023-07-10T17:08:10.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2023-07-10T17:08:10.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2023-07-10T17:08:10.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2023-07-10T17:08:10.198 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="App mode production" logger=settings 2023-07-10T17:08:10.198 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Initializing SqlStore" logger=server 2023-07-10T17:08:10.198 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2023-07-10T17:08:10.198 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Starting DB migration" logger=migrator 2023-07-10T17:08:10.198 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2023-07-10T17:08:10.198 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2023-07-10T17:08:10.198 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2023-07-10T17:08:10.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2023-07-10T17:08:10.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2023-07-10T17:08:10.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2023-07-10T17:08:10.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2023-07-10T17:08:10.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2023-07-10T17:08:10.774 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2023-07-10T17:08:10.774 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2023-07-10T17:08:10.774 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2023-07-10T17:08:10.774 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2023-07-10T17:08:10.775 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[23730]: audit 2023-07-10T17:08:08.883857+0000 mgr.x (mgr.24857) 94 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:08:10.775 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[23730]: audit 2023-07-10T17:08:08.885477+0000 mgr.x (mgr.24857) 95 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi116.front.sepia.ceph.com:9093"}]: dispatch 2023-07-10T17:08:10.775 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[23730]: audit 2023-07-10T17:08:08.906388+0000 mgr.x (mgr.24857) 96 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:08:10.775 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[23730]: audit 2023-07-10T17:08:08.926410+0000 mgr.x (mgr.24857) 97 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:08:10.775 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[23730]: audit 2023-07-10T17:08:08.936498+0000 mgr.x (mgr.24857) 98 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:08:10.775 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[23730]: audit 2023-07-10T17:08:08.939135+0000 mgr.x (mgr.24857) 99 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2023-07-10T17:08:10.776 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[23730]: audit 2023-07-10T17:08:08.973196+0000 mgr.x (mgr.24857) 100 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:08:10.776 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[23730]: audit 2023-07-10T17:08:08.977973+0000 mgr.x (mgr.24857) 101 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2023-07-10T17:08:10.776 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[23730]: cephadm 2023-07-10T17:08:09.031472+0000 mgr.x (mgr.24857) 102 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:08:10.776 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[23730]: cephadm 2023-07-10T17:08:09.032983+0000 mgr.x (mgr.24857) 103 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:08:10.850 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[28351]: audit 2023-07-10T17:08:08.883857+0000 mgr.x (mgr.24857) 94 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:08:10.851 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[28351]: audit 2023-07-10T17:08:08.885477+0000 mgr.x (mgr.24857) 95 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi116.front.sepia.ceph.com:9093"}]: dispatch 2023-07-10T17:08:10.851 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[28351]: audit 2023-07-10T17:08:08.906388+0000 mgr.x (mgr.24857) 96 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:08:10.851 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[28351]: audit 2023-07-10T17:08:08.926410+0000 mgr.x (mgr.24857) 97 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:08:10.851 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[28351]: audit 2023-07-10T17:08:08.936498+0000 mgr.x (mgr.24857) 98 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:08:10.851 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[28351]: audit 2023-07-10T17:08:08.939135+0000 mgr.x (mgr.24857) 99 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2023-07-10T17:08:10.851 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[28351]: audit 2023-07-10T17:08:08.973196+0000 mgr.x (mgr.24857) 100 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:08:10.851 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[28351]: audit 2023-07-10T17:08:08.977973+0000 mgr.x (mgr.24857) 101 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2023-07-10T17:08:10.852 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[28351]: cephadm 2023-07-10T17:08:09.031472+0000 mgr.x (mgr.24857) 102 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:08:10.852 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[28351]: cephadm 2023-07-10T17:08:09.032983+0000 mgr.x (mgr.24857) 103 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:08:10.852 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[23740]: audit 2023-07-10T17:08:08.883857+0000 mgr.x (mgr.24857) 94 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:08:10.852 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[23740]: audit 2023-07-10T17:08:08.885477+0000 mgr.x (mgr.24857) 95 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi116.front.sepia.ceph.com:9093"}]: dispatch 2023-07-10T17:08:10.852 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[23740]: audit 2023-07-10T17:08:08.906388+0000 mgr.x (mgr.24857) 96 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:08:10.852 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[23740]: audit 2023-07-10T17:08:08.926410+0000 mgr.x (mgr.24857) 97 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:08:10.853 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[23740]: audit 2023-07-10T17:08:08.936498+0000 mgr.x (mgr.24857) 98 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:08:10.853 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[23740]: audit 2023-07-10T17:08:08.939135+0000 mgr.x (mgr.24857) 99 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2023-07-10T17:08:10.853 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[23740]: audit 2023-07-10T17:08:08.973196+0000 mgr.x (mgr.24857) 100 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:08:10.853 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[23740]: audit 2023-07-10T17:08:08.977973+0000 mgr.x (mgr.24857) 101 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2023-07-10T17:08:10.853 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[23740]: cephadm 2023-07-10T17:08:09.031472+0000 mgr.x (mgr.24857) 102 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:08:10.853 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:10 smithi116 bash[23740]: cephadm 2023-07-10T17:08:09.032983+0000 mgr.x (mgr.24857) 103 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:08:11.059 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2023-07-10T17:08:11.059 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2023-07-10T17:08:11.059 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2023-07-10T17:08:11.060 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:10 smithi192 bash[45347]: t=2023-07-10T17:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2023-07-10T17:08:11.335 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[45347]: t=2023-07-10T17:08:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2023-07-10T17:08:11.336 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[45347]: t=2023-07-10T17:08:11+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2023-07-10T17:08:11.336 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[45347]: t=2023-07-10T17:08:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2023-07-10T17:08:11.336 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[45347]: t=2023-07-10T17:08:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2023-07-10T17:08:11.336 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[45347]: t=2023-07-10T17:08:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2023-07-10T17:08:11.612 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[45347]: t=2023-07-10T17:08:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2023-07-10T17:08:11.612 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[45347]: t=2023-07-10T17:08:11+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2023-07-10T17:08:11.613 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[45347]: t=2023-07-10T17:08:11+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2023-07-10T17:08:11.864 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[23730]: cluster 2023-07-10T17:08:10.015243+0000 mgr.x (mgr.24857) 104 : cluster [DBG] pgmap v61: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:11.864 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[23730]: cephadm 2023-07-10T17:08:11.403617+0000 mgr.x (mgr.24857) 105 : cephadm [INF] Upgrade: Updating mgr.y 2023-07-10T17:08:11.864 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[23730]: audit 2023-07-10T17:08:11.413373+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:11.865 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[23730]: audit 2023-07-10T17:08:11.416672+0000 mon.b (mon.2) 70 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:08:11.865 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[23730]: audit 2023-07-10T17:08:11.416996+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:08:11.865 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[23730]: audit 2023-07-10T17:08:11.419768+0000 mon.b (mon.2) 71 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T17:08:11.865 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[23730]: audit 2023-07-10T17:08:11.421513+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:08:11.865 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[23730]: cephadm 2023-07-10T17:08:11.423499+0000 mgr.x (mgr.24857) 106 : cephadm [INF] Deploying daemon mgr.y on smithi116 2023-07-10T17:08:11.866 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[45347]: t=2023-07-10T17:08:11+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2023-07-10T17:08:11.866 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[45347]: t=2023-07-10T17:08:11+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2023-07-10T17:08:11.866 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[45347]: t=2023-07-10T17:08:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2023-07-10T17:08:11.866 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[45347]: t=2023-07-10T17:08:11+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2023-07-10T17:08:11.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:11 smithi116 bash[28351]: cluster 2023-07-10T17:08:10.015243+0000 mgr.x (mgr.24857) 104 : cluster [DBG] pgmap v61: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:11.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:11 smithi116 bash[28351]: cephadm 2023-07-10T17:08:11.403617+0000 mgr.x (mgr.24857) 105 : cephadm [INF] Upgrade: Updating mgr.y 2023-07-10T17:08:11.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:11 smithi116 bash[28351]: audit 2023-07-10T17:08:11.413373+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:11.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:11 smithi116 bash[28351]: audit 2023-07-10T17:08:11.416672+0000 mon.b (mon.2) 70 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:08:11.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:11 smithi116 bash[28351]: audit 2023-07-10T17:08:11.416996+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:08:11.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:11 smithi116 bash[28351]: audit 2023-07-10T17:08:11.419768+0000 mon.b (mon.2) 71 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T17:08:11.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:11 smithi116 bash[28351]: audit 2023-07-10T17:08:11.421513+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:08:11.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:11 smithi116 bash[28351]: cephadm 2023-07-10T17:08:11.423499+0000 mgr.x (mgr.24857) 106 : cephadm [INF] Deploying daemon mgr.y on smithi116 2023-07-10T17:08:11.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:11 smithi116 bash[23740]: cluster 2023-07-10T17:08:10.015243+0000 mgr.x (mgr.24857) 104 : cluster [DBG] pgmap v61: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:11.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:11 smithi116 bash[23740]: cephadm 2023-07-10T17:08:11.403617+0000 mgr.x (mgr.24857) 105 : cephadm [INF] Upgrade: Updating mgr.y 2023-07-10T17:08:11.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:11 smithi116 bash[23740]: audit 2023-07-10T17:08:11.413373+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:11.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:11 smithi116 bash[23740]: audit 2023-07-10T17:08:11.416672+0000 mon.b (mon.2) 70 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:08:11.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:11 smithi116 bash[23740]: audit 2023-07-10T17:08:11.416996+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:08:11.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:11 smithi116 bash[23740]: audit 2023-07-10T17:08:11.419768+0000 mon.b (mon.2) 71 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T17:08:11.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:11 smithi116 bash[23740]: audit 2023-07-10T17:08:11.421513+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:08:11.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:11 smithi116 bash[23740]: cephadm 2023-07-10T17:08:11.423499+0000 mgr.x (mgr.24857) 106 : cephadm [INF] Deploying daemon mgr.y on smithi116 2023-07-10T17:08:12.136 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:11 smithi192 bash[45347]: t=2023-07-10T17:08:11+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2023-07-10T17:08:12.137 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:12 smithi192 bash[45347]: t=2023-07-10T17:08:12+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2023-07-10T17:08:12.137 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:12 smithi192 bash[45347]: t=2023-07-10T17:08:12+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2023-07-10T17:08:12.483 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:12 smithi192 bash[45347]: t=2023-07-10T17:08:12+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2023-07-10T17:08:12.483 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:12 smithi192 bash[45347]: t=2023-07-10T17:08:12+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2023-07-10T17:08:12.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:12 smithi192 bash[45347]: t=2023-07-10T17:08:12+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2023-07-10T17:08:12.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:12 smithi192 bash[45347]: t=2023-07-10T17:08:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2023-07-10T17:08:12.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:12 smithi192 bash[45347]: t=2023-07-10T17:08:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2023-07-10T17:08:12.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:12 smithi192 bash[45347]: t=2023-07-10T17:08:12+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2023-07-10T17:08:12.786 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:12 smithi192 bash[23730]: cluster 2023-07-10T17:08:12.016947+0000 mgr.x (mgr.24857) 107 : cluster [DBG] pgmap v62: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:12.786 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:12 smithi192 bash[45347]: t=2023-07-10T17:08:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2023-07-10T17:08:12.786 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:12 smithi192 bash[45347]: t=2023-07-10T17:08:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2023-07-10T17:08:12.787 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:12 smithi192 bash[45347]: t=2023-07-10T17:08:12+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2023-07-10T17:08:12.787 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:12 smithi192 bash[45347]: t=2023-07-10T17:08:12+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2023-07-10T17:08:12.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:12 smithi116 bash[23740]: cluster 2023-07-10T17:08:12.016947+0000 mgr.x (mgr.24857) 107 : cluster [DBG] pgmap v62: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:12.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:12 smithi116 bash[28351]: cluster 2023-07-10T17:08:12.016947+0000 mgr.x (mgr.24857) 107 : cluster [DBG] pgmap v62: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:08:13.071 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:12 smithi192 bash[45347]: t=2023-07-10T17:08:12+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2023-07-10T17:08:13.071 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:12 smithi192 bash[45347]: t=2023-07-10T17:08:12+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2023-07-10T17:08:13.071 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:13 smithi192 bash[45347]: t=2023-07-10T17:08:13+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2023-07-10T17:08:13.356 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:13 smithi192 bash[45347]: t=2023-07-10T17:08:13+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2023-07-10T17:08:13.356 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:13 smithi192 bash[45347]: t=2023-07-10T17:08:13+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2023-07-10T17:08:13.357 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:13 smithi192 bash[45347]: t=2023-07-10T17:08:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2023-07-10T17:08:13.357 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:13 smithi192 bash[45347]: t=2023-07-10T17:08:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2023-07-10T17:08:13.357 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:13 smithi192 bash[45347]: t=2023-07-10T17:08:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2023-07-10T17:08:13.358 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:13 smithi192 bash[45347]: t=2023-07-10T17:08:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2023-07-10T17:08:13.624 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:13 smithi192 bash[45347]: t=2023-07-10T17:08:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2023-07-10T17:08:13.624 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:13 smithi192 bash[45347]: t=2023-07-10T17:08:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2023-07-10T17:08:13.625 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:13 smithi192 bash[45347]: t=2023-07-10T17:08:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2023-07-10T17:08:13.875 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:13 smithi192 bash[45347]: t=2023-07-10T17:08:13+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2023-07-10T17:08:13.876 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:13 smithi192 bash[45347]: t=2023-07-10T17:08:13+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2023-07-10T17:08:13.876 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:13 smithi192 bash[45347]: t=2023-07-10T17:08:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2023-07-10T17:08:13.876 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:13 smithi192 bash[45347]: t=2023-07-10T17:08:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2023-07-10T17:08:13.876 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:13 smithi192 bash[45347]: t=2023-07-10T17:08:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2023-07-10T17:08:13.893 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:13 smithi116 systemd[1]: Stopping Ceph mgr.y for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:08:14.135 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:13 smithi192 bash[45347]: t=2023-07-10T17:08:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2023-07-10T17:08:14.135 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:14 smithi192 bash[45347]: t=2023-07-10T17:08:14+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2023-07-10T17:08:14.136 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:14 smithi192 bash[45347]: t=2023-07-10T17:08:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2023-07-10T17:08:14.386 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:14 smithi192 bash[45347]: t=2023-07-10T17:08:14+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2023-07-10T17:08:14.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:14 smithi192 bash[45347]: t=2023-07-10T17:08:14+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2023-07-10T17:08:14.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:14 smithi192 bash[45347]: t=2023-07-10T17:08:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2023-07-10T17:08:14.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:14 smithi192 bash[45347]: t=2023-07-10T17:08:14+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2023-07-10T17:08:14.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:14 smithi192 bash[45347]: t=2023-07-10T17:08:14+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2023-07-10T17:08:14.521 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:14 smithi192 bash[45347]: t=2023-07-10T17:08:14+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2023-07-10T17:08:14.731 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:14 smithi192 bash[45347]: t=2023-07-10T17:08:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2023-07-10T17:08:14.731 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:14 smithi192 bash[45347]: t=2023-07-10T17:08:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2023-07-10T17:08:14.732 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:14 smithi192 bash[45347]: t=2023-07-10T17:08:14+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2023-07-10T17:08:14.999 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:14 smithi192 bash[45347]: t=2023-07-10T17:08:14+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2023-07-10T17:08:14.999 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:14 smithi192 bash[45347]: t=2023-07-10T17:08:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2023-07-10T17:08:15.000 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:14 smithi192 bash[45347]: t=2023-07-10T17:08:14+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2023-07-10T17:08:15.000 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:14 smithi192 bash[45347]: t=2023-07-10T17:08:14+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2023-07-10T17:08:15.000 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:14 smithi192 bash[45347]: t=2023-07-10T17:08:14+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2023-07-10T17:08:15.043 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:14 smithi116 bash[59232]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-mgr.y 2023-07-10T17:08:15.044 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:14 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-10T17:08:15.044 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:14 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.y.service: Failed with result 'exit-code'. 2023-07-10T17:08:15.044 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:14 smithi116 systemd[1]: Stopped Ceph mgr.y for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:08:15.044 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:15 smithi116 systemd[1]: Started Ceph mgr.y for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:08:15.284 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:15 smithi192 bash[45347]: t=2023-07-10T17:08:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2023-07-10T17:08:15.284 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:15 smithi192 bash[45347]: t=2023-07-10T17:08:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2023-07-10T17:08:15.285 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:15 smithi192 bash[45347]: t=2023-07-10T17:08:15+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2023-07-10T17:08:15.589 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:15 smithi192 bash[45347]: t=2023-07-10T17:08:15+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2023-07-10T17:08:15.589 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:15 smithi192 bash[45347]: t=2023-07-10T17:08:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2023-07-10T17:08:15.589 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:15 smithi192 bash[45347]: t=2023-07-10T17:08:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2023-07-10T17:08:15.590 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:15 smithi192 bash[45347]: t=2023-07-10T17:08:15+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2023-07-10T17:08:15.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:15 smithi192 bash[23730]: cluster 2023-07-10T17:08:14.017814+0000 mgr.x (mgr.24857) 108 : cluster [DBG] pgmap v63: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:15.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:15 smithi192 bash[23730]: audit 2023-07-10T17:08:15.087525+0000 mon.a (mon.0) 731 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:15.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:15 smithi192 bash[23730]: audit 2023-07-10T17:08:15.099398+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:15.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:15 smithi192 bash[45347]: t=2023-07-10T17:08:15+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2023-07-10T17:08:15.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:15 smithi192 bash[45347]: t=2023-07-10T17:08:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2023-07-10T17:08:15.888 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:15 smithi192 bash[45347]: t=2023-07-10T17:08:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2023-07-10T17:08:15.888 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:15 smithi192 bash[45347]: t=2023-07-10T17:08:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2023-07-10T17:08:15.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:15 smithi116 bash[28351]: cluster 2023-07-10T17:08:14.017814+0000 mgr.x (mgr.24857) 108 : cluster [DBG] pgmap v63: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:15.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:15 smithi116 bash[28351]: audit 2023-07-10T17:08:15.087525+0000 mon.a (mon.0) 731 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:15.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:15 smithi116 bash[28351]: audit 2023-07-10T17:08:15.099398+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:15.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:15 smithi116 bash[23740]: cluster 2023-07-10T17:08:14.017814+0000 mgr.x (mgr.24857) 108 : cluster [DBG] pgmap v63: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:15.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:15 smithi116 bash[23740]: audit 2023-07-10T17:08:15.087525+0000 mon.a (mon.0) 731 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:15.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:15 smithi116 bash[23740]: audit 2023-07-10T17:08:15.099398+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:16.156 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:15 smithi192 bash[45347]: t=2023-07-10T17:08:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2023-07-10T17:08:16.156 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:15 smithi192 bash[45347]: t=2023-07-10T17:08:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2023-07-10T17:08:16.157 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:16 smithi192 bash[45347]: t=2023-07-10T17:08:16+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2023-07-10T17:08:16.157 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:16 smithi192 bash[45347]: t=2023-07-10T17:08:16+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2023-07-10T17:08:16.442 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:16 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 5. 2023-07-10T17:08:16.442 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:16 smithi192 systemd[1]: Stopped Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:08:16.442 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:16 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Start request repeated too quickly. 2023-07-10T17:08:16.443 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:16 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-10T17:08:16.443 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:16 smithi192 systemd[1]: Failed to start Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:08:16.443 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:16 smithi192 bash[45347]: t=2023-07-10T17:08:16+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2023-07-10T17:08:16.444 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:16 smithi192 bash[45347]: t=2023-07-10T17:08:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2023-07-10T17:08:16.444 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:16 smithi192 bash[45347]: t=2023-07-10T17:08:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2023-07-10T17:08:16.444 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:16 smithi192 bash[45347]: t=2023-07-10T17:08:16+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2023-07-10T17:08:16.444 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:16 smithi192 bash[45347]: t=2023-07-10T17:08:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2023-07-10T17:08:16.444 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:16 smithi192 bash[45347]: t=2023-07-10T17:08:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2023-07-10T17:08:16.558 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:16 smithi116 bash[59394]: debug 2023-07-10T17:08:16.383+0000 7f384d0e2700 1 -- 172.21.15.116:0/2169212990 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5614d723b1e0 con 0x5614d7fbac00 2023-07-10T17:08:16.760 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:16 smithi192 bash[45347]: t=2023-07-10T17:08:16+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2023-07-10T17:08:16.760 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:16 smithi192 bash[45347]: t=2023-07-10T17:08:16+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2023-07-10T17:08:16.760 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:16 smithi192 bash[45347]: t=2023-07-10T17:08:16+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2023-07-10T17:08:16.761 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:16 smithi192 bash[45347]: t=2023-07-10T17:08:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2023-07-10T17:08:16.761 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:16 smithi192 bash[45347]: t=2023-07-10T17:08:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2023-07-10T17:08:16.893 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:16 smithi116 bash[59394]: debug 2023-07-10T17:08:16.551+0000 7f3857230200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-10T17:08:17.020 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:16 smithi192 bash[45347]: t=2023-07-10T17:08:16+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2023-07-10T17:08:17.020 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:16 smithi192 bash[45347]: t=2023-07-10T17:08:16+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2023-07-10T17:08:17.020 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:16 smithi192 bash[45347]: t=2023-07-10T17:08:16+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2023-07-10T17:08:17.020 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:17 smithi192 bash[45347]: t=2023-07-10T17:08:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2023-07-10T17:08:17.277 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:17 smithi192 bash[45347]: t=2023-07-10T17:08:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2023-07-10T17:08:17.277 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:17 smithi192 bash[45347]: t=2023-07-10T17:08:17+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2023-07-10T17:08:17.564 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:17 smithi192 bash[45347]: t=2023-07-10T17:08:17+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2023-07-10T17:08:17.565 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:17 smithi192 bash[45347]: t=2023-07-10T17:08:17+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2023-07-10T17:08:17.565 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:17 smithi192 bash[45347]: t=2023-07-10T17:08:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2023-07-10T17:08:17.565 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:17 smithi192 bash[45347]: t=2023-07-10T17:08:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2023-07-10T17:08:17.565 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:17 smithi192 bash[45347]: t=2023-07-10T17:08:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2023-07-10T17:08:17.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:17 smithi192 bash[23730]: cluster 2023-07-10T17:08:16.019418+0000 mgr.x (mgr.24857) 109 : cluster [DBG] pgmap v64: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:17.825 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:17 smithi192 bash[45347]: t=2023-07-10T17:08:17+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2023-07-10T17:08:17.825 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:17 smithi192 bash[45347]: t=2023-07-10T17:08:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2023-07-10T17:08:17.826 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:17 smithi192 bash[45347]: t=2023-07-10T17:08:17+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2023-07-10T17:08:17.826 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:17 smithi192 bash[45347]: t=2023-07-10T17:08:17+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2023-07-10T17:08:17.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:17 smithi116 bash[23740]: cluster 2023-07-10T17:08:16.019418+0000 mgr.x (mgr.24857) 109 : cluster [DBG] pgmap v64: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:17.881 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:17 smithi116 bash[28351]: cluster 2023-07-10T17:08:16.019418+0000 mgr.x (mgr.24857) 109 : cluster [DBG] pgmap v64: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:17.881 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:17 smithi116 bash[59394]: debug 2023-07-10T17:08:17.411+0000 7f3857230200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-10T17:08:17.881 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:17 smithi116 bash[59394]: debug 2023-07-10T17:08:17.515+0000 7f3857230200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-10T17:08:17.881 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:17 smithi116 bash[59394]: debug 2023-07-10T17:08:17.615+0000 7f3857230200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-10T17:08:18.076 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:17 smithi192 bash[45347]: t=2023-07-10T17:08:17+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2023-07-10T17:08:18.076 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:17 smithi192 bash[45347]: t=2023-07-10T17:08:17+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2023-07-10T17:08:18.076 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:17 smithi192 bash[45347]: t=2023-07-10T17:08:17+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2023-07-10T17:08:18.077 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:18 smithi192 bash[45347]: t=2023-07-10T17:08:18+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2023-07-10T17:08:18.143 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:17 smithi116 bash[59394]: debug 2023-07-10T17:08:17.871+0000 7f3857230200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-10T17:08:18.344 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:18 smithi192 bash[45347]: t=2023-07-10T17:08:18+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2023-07-10T17:08:18.344 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:18 smithi192 bash[45347]: t=2023-07-10T17:08:18+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2023-07-10T17:08:18.344 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:18 smithi192 bash[45347]: t=2023-07-10T17:08:18+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2023-07-10T17:08:18.598 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:18 smithi116 bash[59394]: debug 2023-07-10T17:08:18.219+0000 7f3857230200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-10T17:08:18.599 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:18 smithi116 bash[59394]: debug 2023-07-10T17:08:18.375+0000 7f3857230200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-10T17:08:18.636 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:18 smithi192 bash[45347]: t=2023-07-10T17:08:18+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2023-07-10T17:08:18.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:18 smithi192 bash[45347]: t=2023-07-10T17:08:18+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2023-07-10T17:08:18.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:18 smithi192 bash[45347]: t=2023-07-10T17:08:18+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2023-07-10T17:08:18.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:18 smithi192 bash[45347]: t=2023-07-10T17:08:18+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2023-07-10T17:08:18.889 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:18 smithi192 bash[45347]: t=2023-07-10T17:08:18+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2023-07-10T17:08:18.889 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:18 smithi192 bash[45347]: t=2023-07-10T17:08:18+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2023-07-10T17:08:18.889 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:18 smithi192 bash[45347]: t=2023-07-10T17:08:18+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2023-07-10T17:08:18.890 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:18 smithi192 bash[45347]: t=2023-07-10T17:08:18+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2023-07-10T17:08:18.890 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:18 smithi192 bash[45347]: t=2023-07-10T17:08:18+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2023-07-10T17:08:18.892 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:18 smithi116 bash[59394]: debug 2023-07-10T17:08:18.591+0000 7f3857230200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-10T17:08:19.216 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:18 smithi192 bash[45347]: t=2023-07-10T17:08:18+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2023-07-10T17:08:19.216 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[45347]: t=2023-07-10T17:08:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2023-07-10T17:08:19.216 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[45347]: t=2023-07-10T17:08:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2023-07-10T17:08:19.216 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[45347]: t=2023-07-10T17:08:19+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2023-07-10T17:08:19.217 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[45347]: t=2023-07-10T17:08:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2023-07-10T17:08:19.501 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[45347]: t=2023-07-10T17:08:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2023-07-10T17:08:19.501 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[45347]: t=2023-07-10T17:08:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2023-07-10T17:08:19.502 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[45347]: t=2023-07-10T17:08:19+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2023-07-10T17:08:19.502 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[45347]: t=2023-07-10T17:08:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2023-07-10T17:08:19.502 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[45347]: t=2023-07-10T17:08:19+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2023-07-10T17:08:19.740 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:08:19.769 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[23730]: cluster 2023-07-10T17:08:18.020097+0000 mgr.x (mgr.24857) 110 : cluster [DBG] pgmap v65: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:19.770 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[45347]: t=2023-07-10T17:08:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2023-07-10T17:08:19.770 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[45347]: t=2023-07-10T17:08:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2023-07-10T17:08:19.770 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[45347]: t=2023-07-10T17:08:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2023-07-10T17:08:19.893 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:19 smithi116 bash[59394]: debug 2023-07-10T17:08:19.591+0000 7f3857230200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-10T17:08:19.893 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:19 smithi116 bash[59394]: debug 2023-07-10T17:08:19.703+0000 7f3857230200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-10T17:08:19.893 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:19 smithi116 bash[59394]: debug 2023-07-10T17:08:19.823+0000 7f3857230200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-10T17:08:19.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:19 smithi116 bash[23740]: cluster 2023-07-10T17:08:18.020097+0000 mgr.x (mgr.24857) 110 : cluster [DBG] pgmap v65: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:19.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:19 smithi116 bash[28351]: cluster 2023-07-10T17:08:18.020097+0000 mgr.x (mgr.24857) 110 : cluster [DBG] pgmap v65: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:20.030 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[45347]: t=2023-07-10T17:08:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2023-07-10T17:08:20.030 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[45347]: t=2023-07-10T17:08:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2023-07-10T17:08:20.030 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[45347]: t=2023-07-10T17:08:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2023-07-10T17:08:20.030 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:19 smithi192 bash[45347]: t=2023-07-10T17:08:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2023-07-10T17:08:20.030 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:20 smithi192 bash[45347]: t=2023-07-10T17:08:20+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2023-07-10T17:08:20.298 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:20 smithi192 bash[45347]: t=2023-07-10T17:08:20+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2023-07-10T17:08:20.298 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:20 smithi192 bash[45347]: t=2023-07-10T17:08:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2023-07-10T17:08:20.299 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:20 smithi192 bash[45347]: t=2023-07-10T17:08:20+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2023-07-10T17:08:20.536 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:08:20.536 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 starting - - - - 2023-07-10T17:08:20.536 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 starting - - - - 2023-07-10T17:08:20.537 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 starting - - - - 2023-07-10T17:08:20.537 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443 running (2m) 111s ago 12m 460M - 18.1.2-30-g8507faa7 17e8f0d79247 d22a6b7f47df 2023-07-10T17:08:20.537 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 starting - - - - 2023-07-10T17:08:20.537 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (13m) 111s ago 14m 58.6M 2048M 16.2.0 24ecd6d5f14c 33f26d450a67 2023-07-10T17:08:20.537 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (12m) 111s ago 12m 52.3M 2048M 16.2.0 24ecd6d5f14c 9ad4329e5488 2023-07-10T17:08:20.537 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (12m) 111s ago 12m 48.4M 2048M 16.2.0 24ecd6d5f14c 31e3ac12741b 2023-07-10T17:08:20.537 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 starting - - - - 2023-07-10T17:08:20.538 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 starting - - - - 2023-07-10T17:08:20.538 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (11m) 111s ago 11m 52.8M 4096M 16.2.0 24ecd6d5f14c 4fa26367c830 2023-07-10T17:08:20.538 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (11m) 111s ago 11m 52.0M 4096M 16.2.0 24ecd6d5f14c c157509b3fd4 2023-07-10T17:08:20.538 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (10m) 111s ago 10m 48.1M 4096M 16.2.0 24ecd6d5f14c f1fd77b3c454 2023-07-10T17:08:20.538 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (10m) 111s ago 10m 53.4M 4096M 16.2.0 24ecd6d5f14c 60949d479c11 2023-07-10T17:08:20.538 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (10m) 111s ago 10m 50.9M 4096M 16.2.0 24ecd6d5f14c 75fc0b36aab8 2023-07-10T17:08:20.538 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (9m) 111s ago 9m 49.2M 4096M 16.2.0 24ecd6d5f14c d673065333dd 2023-07-10T17:08:20.538 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (9m) 111s ago 9m 49.4M 4096M 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:08:20.539 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (8m) 111s ago 8m 52.7M 4096M 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:08:20.539 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 starting - - - - 2023-07-10T17:08:20.539 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (5m) 111s ago 5m 52.8M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:08:20.539 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (5m) 111s ago 5m 52.6M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:08:20.539 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (5m) 111s ago 5m 53.0M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:08:20.539 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (5m) 111s ago 5m 54.6M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:08:20.595 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:20 smithi116 bash[59394]: debug 2023-07-10T17:08:20.355+0000 7f3857230200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-10T17:08:20.630 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:20 smithi192 bash[45347]: t=2023-07-10T17:08:20+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2023-07-10T17:08:20.630 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:20 smithi192 bash[45347]: t=2023-07-10T17:08:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2023-07-10T17:08:20.631 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:20 smithi192 bash[45347]: t=2023-07-10T17:08:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2023-07-10T17:08:20.631 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:20 smithi192 bash[45347]: t=2023-07-10T17:08:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2023-07-10T17:08:20.631 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:20 smithi192 bash[45347]: t=2023-07-10T17:08:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2023-07-10T17:08:20.631 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:20 smithi192 bash[45347]: t=2023-07-10T17:08:20+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2023-07-10T17:08:20.886 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:20 smithi192 bash[45347]: t=2023-07-10T17:08:20+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2023-07-10T17:08:20.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:20 smithi192 bash[45347]: t=2023-07-10T17:08:20+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2023-07-10T17:08:20.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:20 smithi192 bash[45347]: t=2023-07-10T17:08:20+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2023-07-10T17:08:20.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:20 smithi192 bash[45347]: t=2023-07-10T17:08:20+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2023-07-10T17:08:20.893 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:20 smithi116 bash[59394]: debug 2023-07-10T17:08:20.587+0000 7f3857230200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-10T17:08:20.998 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:08:20.998 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:08:20.998 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2023-07-10T17:08:20.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:08:20.999 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:08:20.999 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2023-07-10T17:08:20.999 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 1 2023-07-10T17:08:20.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:08:20.999 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:08:21.000 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-10T17:08:21.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:08:21.000 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": {}, 2023-07-10T17:08:21.000 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:08:21.000 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:08:21.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:08:21.001 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:08:21.001 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 16, 2023-07-10T17:08:21.001 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 1 2023-07-10T17:08:21.001 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:08:21.001 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:08:21.229 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:20 smithi192 bash[45347]: t=2023-07-10T17:08:20+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2023-07-10T17:08:21.230 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[45347]: t=2023-07-10T17:08:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2023-07-10T17:08:21.230 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[45347]: t=2023-07-10T17:08:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2023-07-10T17:08:21.230 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[45347]: t=2023-07-10T17:08:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2023-07-10T17:08:21.230 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[45347]: t=2023-07-10T17:08:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2023-07-10T17:08:21.392 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:21 smithi116 bash[59394]: debug 2023-07-10T17:08:21.023+0000 7f3857230200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-10T17:08:21.396 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:08:21.396 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:08:21.396 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:08:21.396 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:08:21.396 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [], 2023-07-10T17:08:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "", 2023-07-10T17:08:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading mgr daemons", 2023-07-10T17:08:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:08:21.397 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:08:21.506 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[45347]: t=2023-07-10T17:08:21+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2023-07-10T17:08:21.507 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[45347]: t=2023-07-10T17:08:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2023-07-10T17:08:21.507 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[45347]: t=2023-07-10T17:08:21+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2023-07-10T17:08:21.834 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[23730]: audit 2023-07-10T17:08:19.722434+0000 mgr.x (mgr.24857) 111 : audit [DBG] from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:21.834 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[23730]: cluster 2023-07-10T17:08:20.020950+0000 mgr.x (mgr.24857) 112 : cluster [DBG] pgmap v66: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:21.834 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[23730]: audit 2023-07-10T17:08:20.100263+0000 mgr.x (mgr.24857) 113 : audit [DBG] from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:21.834 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[23730]: audit 2023-07-10T17:08:20.507470+0000 mgr.x (mgr.24857) 114 : audit [DBG] from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:21.834 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[23730]: audit 2023-07-10T17:08:20.994270+0000 mon.a (mon.0) 733 : audit [DBG] from='client.? 172.21.15.116:0/3479901191' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:08:21.835 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[45347]: t=2023-07-10T17:08:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2023-07-10T17:08:21.835 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[45347]: t=2023-07-10T17:08:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2023-07-10T17:08:21.835 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[45347]: t=2023-07-10T17:08:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2023-07-10T17:08:21.835 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[45347]: t=2023-07-10T17:08:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2023-07-10T17:08:21.835 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[45347]: t=2023-07-10T17:08:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2023-07-10T17:08:21.849 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2023-07-10T17:08:21.850 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2023-07-10T17:08:21.850 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:08:21.893 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:21 smithi116 bash[59394]: debug 2023-07-10T17:08:21.623+0000 7f3857230200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-10T17:08:21.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:21 smithi116 bash[28351]: audit 2023-07-10T17:08:19.722434+0000 mgr.x (mgr.24857) 111 : audit [DBG] from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:21.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:21 smithi116 bash[28351]: cluster 2023-07-10T17:08:20.020950+0000 mgr.x (mgr.24857) 112 : cluster [DBG] pgmap v66: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:21.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:21 smithi116 bash[28351]: audit 2023-07-10T17:08:20.100263+0000 mgr.x (mgr.24857) 113 : audit [DBG] from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:21.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:21 smithi116 bash[28351]: audit 2023-07-10T17:08:20.507470+0000 mgr.x (mgr.24857) 114 : audit [DBG] from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:21.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:21 smithi116 bash[28351]: audit 2023-07-10T17:08:20.994270+0000 mon.a (mon.0) 733 : audit [DBG] from='client.? 172.21.15.116:0/3479901191' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:08:21.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:21 smithi116 bash[23740]: audit 2023-07-10T17:08:19.722434+0000 mgr.x (mgr.24857) 111 : audit [DBG] from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:21.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:21 smithi116 bash[23740]: cluster 2023-07-10T17:08:20.020950+0000 mgr.x (mgr.24857) 112 : cluster [DBG] pgmap v66: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:21.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:21 smithi116 bash[23740]: audit 2023-07-10T17:08:20.100263+0000 mgr.x (mgr.24857) 113 : audit [DBG] from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:21.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:21 smithi116 bash[23740]: audit 2023-07-10T17:08:20.507470+0000 mgr.x (mgr.24857) 114 : audit [DBG] from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:21.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:21 smithi116 bash[23740]: audit 2023-07-10T17:08:20.994270+0000 mon.a (mon.0) 733 : audit [DBG] from='client.? 172.21.15.116:0/3479901191' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:08:22.122 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[45347]: t=2023-07-10T17:08:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2023-07-10T17:08:22.123 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:21 smithi192 bash[45347]: t=2023-07-10T17:08:21+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2023-07-10T17:08:22.123 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[45347]: t=2023-07-10T17:08:22+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2023-07-10T17:08:22.123 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[45347]: t=2023-07-10T17:08:22+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2023-07-10T17:08:22.374 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[45347]: t=2023-07-10T17:08:22+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2023-07-10T17:08:22.374 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[45347]: t=2023-07-10T17:08:22+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2023-07-10T17:08:22.375 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[45347]: t=2023-07-10T17:08:22+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2023-07-10T17:08:22.375 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[45347]: t=2023-07-10T17:08:22+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2023-07-10T17:08:22.606 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[59394]: debug 2023-07-10T17:08:22.223+0000 7f3857230200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-10T17:08:22.607 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[59394]: debug 2023-07-10T17:08:22.335+0000 7f3857230200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-10T17:08:22.636 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[45347]: t=2023-07-10T17:08:22+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2023-07-10T17:08:22.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[45347]: t=2023-07-10T17:08:22+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2023-07-10T17:08:22.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[45347]: t=2023-07-10T17:08:22+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2023-07-10T17:08:22.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[23730]: audit 2023-07-10T17:08:21.392725+0000 mgr.x (mgr.24857) 115 : audit [DBG] from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:22.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[23730]: audit 2023-07-10T17:08:21.693376+0000 mon.a (mon.0) 734 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[23730]: audit 2023-07-10T17:08:21.701955+0000 mon.a (mon.0) 735 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[23730]: audit 2023-07-10T17:08:21.755995+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[23730]: audit 2023-07-10T17:08:21.764248+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[23730]: audit 2023-07-10T17:08:21.846405+0000 mon.c (mon.1) 51 : audit [DBG] from='client.? 172.21.15.116:0/2927014068' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:08:22.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[23730]: audit 2023-07-10T17:08:22.564909+0000 mon.a (mon.0) 738 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[23730]: audit 2023-07-10T17:08:22.578366+0000 mon.a (mon.0) 739 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.893 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[59394]: debug 2023-07-10T17:08:22.691+0000 7f3857230200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-10T17:08:22.893 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[59394]: debug 2023-07-10T17:08:22.811+0000 7f3857230200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-10T17:08:22.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[23740]: audit 2023-07-10T17:08:21.392725+0000 mgr.x (mgr.24857) 115 : audit [DBG] from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:22.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[23740]: audit 2023-07-10T17:08:21.693376+0000 mon.a (mon.0) 734 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[23740]: audit 2023-07-10T17:08:21.701955+0000 mon.a (mon.0) 735 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[23740]: audit 2023-07-10T17:08:21.755995+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[23740]: audit 2023-07-10T17:08:21.764248+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[23740]: audit 2023-07-10T17:08:21.846405+0000 mon.c (mon.1) 51 : audit [DBG] from='client.? 172.21.15.116:0/2927014068' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:08:22.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[23740]: audit 2023-07-10T17:08:22.564909+0000 mon.a (mon.0) 738 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[23740]: audit 2023-07-10T17:08:22.578366+0000 mon.a (mon.0) 739 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[28351]: audit 2023-07-10T17:08:21.392725+0000 mgr.x (mgr.24857) 115 : audit [DBG] from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:22.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[28351]: audit 2023-07-10T17:08:21.693376+0000 mon.a (mon.0) 734 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[28351]: audit 2023-07-10T17:08:21.701955+0000 mon.a (mon.0) 735 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[28351]: audit 2023-07-10T17:08:21.755995+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[28351]: audit 2023-07-10T17:08:21.764248+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[28351]: audit 2023-07-10T17:08:21.846405+0000 mon.c (mon.1) 51 : audit [DBG] from='client.? 172.21.15.116:0/2927014068' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:08:22.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[28351]: audit 2023-07-10T17:08:22.564909+0000 mon.a (mon.0) 738 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[28351]: audit 2023-07-10T17:08:22.578366+0000 mon.a (mon.0) 739 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:22.944 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[45347]: t=2023-07-10T17:08:22+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2023-07-10T17:08:22.944 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[45347]: t=2023-07-10T17:08:22+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2023-07-10T17:08:22.945 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[45347]: t=2023-07-10T17:08:22+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2023-07-10T17:08:22.945 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[45347]: t=2023-07-10T17:08:22+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2023-07-10T17:08:22.945 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[45347]: t=2023-07-10T17:08:22+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2023-07-10T17:08:22.945 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:22 smithi192 bash[45347]: t=2023-07-10T17:08:22+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2023-07-10T17:08:23.237 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:23 smithi192 bash[45347]: t=2023-07-10T17:08:23+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2023-07-10T17:08:23.238 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:23 smithi192 bash[45347]: t=2023-07-10T17:08:23+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2023-07-10T17:08:23.238 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:23 smithi192 bash[45347]: t=2023-07-10T17:08:23+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2023-07-10T17:08:23.238 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:23 smithi192 bash[45347]: t=2023-07-10T17:08:23+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2023-07-10T17:08:23.393 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:22 smithi116 bash[59394]: debug 2023-07-10T17:08:22.955+0000 7f3857230200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-10T17:08:23.393 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:23 smithi116 bash[59394]: debug 2023-07-10T17:08:23.071+0000 7f3857230200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-10T17:08:23.393 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:23 smithi116 bash[59394]: debug 2023-07-10T17:08:23.199+0000 7f3857230200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-10T17:08:23.598 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:23 smithi192 bash[45347]: t=2023-07-10T17:08:23+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2023-07-10T17:08:23.598 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:23 smithi192 bash[45347]: t=2023-07-10T17:08:23+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2023-07-10T17:08:23.599 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:23 smithi192 bash[45347]: t=2023-07-10T17:08:23+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2023-07-10T17:08:23.599 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:23 smithi192 bash[45347]: t=2023-07-10T17:08:23+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2023-07-10T17:08:23.599 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:23 smithi192 bash[45347]: t=2023-07-10T17:08:23+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2023-07-10T17:08:23.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:23 smithi192 bash[23730]: cluster 2023-07-10T17:08:22.022568+0000 mgr.x (mgr.24857) 116 : cluster [DBG] pgmap v67: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:23.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:23 smithi192 bash[45347]: t=2023-07-10T17:08:23+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2023-07-10T17:08:23.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:23 smithi192 bash[45347]: t=2023-07-10T17:08:23+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2023-07-10T17:08:23.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:23 smithi192 bash[45347]: t=2023-07-10T17:08:23+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2023-07-10T17:08:23.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:23 smithi192 bash[45347]: t=2023-07-10T17:08:23+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2023-07-10T17:08:23.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:23 smithi116 bash[23740]: cluster 2023-07-10T17:08:22.022568+0000 mgr.x (mgr.24857) 116 : cluster [DBG] pgmap v67: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:23.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:23 smithi116 bash[28351]: cluster 2023-07-10T17:08:22.022568+0000 mgr.x (mgr.24857) 116 : cluster [DBG] pgmap v67: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:23.893 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:23 smithi116 bash[59394]: debug 2023-07-10T17:08:23.383+0000 7f3857230200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-10T17:08:23.893 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:23 smithi116 bash[59394]: debug 2023-07-10T17:08:23.507+0000 7f3857230200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-10T17:08:24.202 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:23 smithi192 bash[45347]: t=2023-07-10T17:08:23+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2023-07-10T17:08:24.203 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:23 smithi192 bash[45347]: t=2023-07-10T17:08:23+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2023-07-10T17:08:24.203 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:24 smithi192 bash[45347]: t=2023-07-10T17:08:24+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2023-07-10T17:08:24.203 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:24 smithi192 bash[45347]: t=2023-07-10T17:08:24+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2023-07-10T17:08:24.203 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:24 smithi192 bash[45347]: t=2023-07-10T17:08:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2023-07-10T17:08:24.203 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:24 smithi192 bash[45347]: t=2023-07-10T17:08:24+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2023-07-10T17:08:24.494 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:24 smithi192 bash[45347]: t=2023-07-10T17:08:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2023-07-10T17:08:24.494 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:24 smithi192 bash[45347]: t=2023-07-10T17:08:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2023-07-10T17:08:24.495 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:24 smithi192 bash[45347]: t=2023-07-10T17:08:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2023-07-10T17:08:24.495 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:24 smithi192 bash[45347]: t=2023-07-10T17:08:24+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2023-07-10T17:08:24.772 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:24 smithi192 bash[45347]: t=2023-07-10T17:08:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2023-07-10T17:08:24.772 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:24 smithi192 bash[45347]: t=2023-07-10T17:08:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2023-07-10T17:08:24.773 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:24 smithi192 bash[45347]: t=2023-07-10T17:08:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2023-07-10T17:08:24.773 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:24 smithi192 bash[45347]: t=2023-07-10T17:08:24+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2023-07-10T17:08:25.040 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:24 smithi192 bash[45347]: t=2023-07-10T17:08:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2023-07-10T17:08:25.040 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:24 smithi192 bash[45347]: t=2023-07-10T17:08:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2023-07-10T17:08:25.041 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:24 smithi192 bash[45347]: t=2023-07-10T17:08:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2023-07-10T17:08:25.041 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2023-07-10T17:08:25.143 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:08:24 smithi116 bash[59394]: debug 2023-07-10T17:08:24.727+0000 7f3857230200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-10T17:08:25.309 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2023-07-10T17:08:25.309 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2023-07-10T17:08:25.310 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2023-07-10T17:08:25.310 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2023-07-10T17:08:25.560 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2023-07-10T17:08:25.561 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2023-07-10T17:08:25.561 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2023-07-10T17:08:25.821 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2023-07-10T17:08:25.821 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2023-07-10T17:08:25.821 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2023-07-10T17:08:25.821 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Created default organization" logger=sqlstore 2023-07-10T17:08:25.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[23730]: cluster 2023-07-10T17:08:24.023273+0000 mgr.x (mgr.24857) 117 : cluster [DBG] pgmap v68: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:25.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[23730]: cluster 2023-07-10T17:08:24.753066+0000 mon.a (mon.0) 740 : cluster [DBG] Standby manager daemon y restarted 2023-07-10T17:08:25.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[23730]: cluster 2023-07-10T17:08:24.753237+0000 mon.a (mon.0) 741 : cluster [DBG] Standby manager daemon y started 2023-07-10T17:08:25.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[23730]: audit 2023-07-10T17:08:24.756696+0000 mon.a (mon.0) 742 : audit [DBG] from='mgr.? 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-10T17:08:25.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[23730]: audit 2023-07-10T17:08:24.757812+0000 mon.a (mon.0) 743 : audit [DBG] from='mgr.? 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:08:25.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[23730]: audit 2023-07-10T17:08:24.759869+0000 mon.a (mon.0) 744 : audit [DBG] from='mgr.? 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-10T17:08:25.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[23730]: audit 2023-07-10T17:08:24.760578+0000 mon.a (mon.0) 745 : audit [DBG] from='mgr.? 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:08:25.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:25 smithi116 bash[23740]: cluster 2023-07-10T17:08:24.023273+0000 mgr.x (mgr.24857) 117 : cluster [DBG] pgmap v68: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:25.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:25 smithi116 bash[23740]: cluster 2023-07-10T17:08:24.753066+0000 mon.a (mon.0) 740 : cluster [DBG] Standby manager daemon y restarted 2023-07-10T17:08:25.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:25 smithi116 bash[23740]: cluster 2023-07-10T17:08:24.753237+0000 mon.a (mon.0) 741 : cluster [DBG] Standby manager daemon y started 2023-07-10T17:08:25.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:25 smithi116 bash[23740]: audit 2023-07-10T17:08:24.756696+0000 mon.a (mon.0) 742 : audit [DBG] from='mgr.? 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-10T17:08:25.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:25 smithi116 bash[23740]: audit 2023-07-10T17:08:24.757812+0000 mon.a (mon.0) 743 : audit [DBG] from='mgr.? 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:08:25.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:25 smithi116 bash[23740]: audit 2023-07-10T17:08:24.759869+0000 mon.a (mon.0) 744 : audit [DBG] from='mgr.? 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-10T17:08:25.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:25 smithi116 bash[23740]: audit 2023-07-10T17:08:24.760578+0000 mon.a (mon.0) 745 : audit [DBG] from='mgr.? 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:08:25.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:25 smithi116 bash[28351]: cluster 2023-07-10T17:08:24.023273+0000 mgr.x (mgr.24857) 117 : cluster [DBG] pgmap v68: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:25.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:25 smithi116 bash[28351]: cluster 2023-07-10T17:08:24.753066+0000 mon.a (mon.0) 740 : cluster [DBG] Standby manager daemon y restarted 2023-07-10T17:08:25.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:25 smithi116 bash[28351]: cluster 2023-07-10T17:08:24.753237+0000 mon.a (mon.0) 741 : cluster [DBG] Standby manager daemon y started 2023-07-10T17:08:25.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:25 smithi116 bash[28351]: audit 2023-07-10T17:08:24.756696+0000 mon.a (mon.0) 742 : audit [DBG] from='mgr.? 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-10T17:08:25.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:25 smithi116 bash[28351]: audit 2023-07-10T17:08:24.757812+0000 mon.a (mon.0) 743 : audit [DBG] from='mgr.? 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:08:25.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:25 smithi116 bash[28351]: audit 2023-07-10T17:08:24.759869+0000 mon.a (mon.0) 744 : audit [DBG] from='mgr.? 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-10T17:08:25.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:25 smithi116 bash[28351]: audit 2023-07-10T17:08:24.760578+0000 mon.a (mon.0) 745 : audit [DBG] from='mgr.? 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:08:26.138 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing HTTPServer" logger=server 2023-07-10T17:08:26.138 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing BackendPluginManager" logger=server 2023-07-10T17:08:26.138 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing PluginManager" logger=server 2023-07-10T17:08:26.138 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Starting plugin search" logger=plugins 2023-07-10T17:08:26.139 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Registering plugin" logger=plugins name="Pie Chart" 2023-07-10T17:08:26.139 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Registering plugin" logger=plugins name="Status Panel" 2023-07-10T17:08:26.139 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing HooksService" logger=server 2023-07-10T17:08:26.139 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing OSSLicensingService" logger=server 2023-07-10T17:08:26.139 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing InternalMetricsService" logger=server 2023-07-10T17:08:26.140 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing RemoteCache" logger=server 2023-07-10T17:08:26.140 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing RenderingService" logger=server 2023-07-10T17:08:26.140 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing AlertEngine" logger=server 2023-07-10T17:08:26.141 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing QuotaService" logger=server 2023-07-10T17:08:26.141 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing ServerLockService" logger=server 2023-07-10T17:08:26.141 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2023-07-10T17:08:26.141 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server 2023-07-10T17:08:26.141 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing LoginService" logger=server 2023-07-10T17:08:26.141 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing SearchService" logger=server 2023-07-10T17:08:26.142 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing TracingService" logger=server 2023-07-10T17:08:26.142 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2023-07-10T17:08:26.142 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing CleanUpService" logger=server 2023-07-10T17:08:26.142 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing NotificationService" logger=server 2023-07-10T17:08:26.142 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2023-07-10T17:08:26.142 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 2023-07-10T17:08:26.142 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:25 smithi192 bash[45347]: t=2023-07-10T17:08:25+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki 2023-07-10T17:08:26.143 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:26 smithi192 bash[45347]: t=2023-07-10T17:08:26+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS 2023-07-10T17:08:26.143 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:26 smithi192 bash[45347]: t=2023-07-10T17:08:26+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS 2023-07-10T17:08:26.143 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:26 smithi192 bash[45347]: t=2023-07-10T17:08:26+0000 lvl=info msg="Initializing Stream Manager" 2023-07-10T17:08:26.143 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:08:26 smithi192 bash[45347]: t=2023-07-10T17:08:26+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2023-07-10T17:08:26.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:26 smithi116 bash[28351]: cluster 2023-07-10T17:08:25.626516+0000 mon.a (mon.0) 746 : cluster [DBG] mgrmap e21: x(active, since 2m), standbys: y 2023-07-10T17:08:26.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:26 smithi116 bash[28351]: cluster 2023-07-10T17:08:26.024611+0000 mgr.x (mgr.24857) 118 : cluster [DBG] pgmap v69: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:26.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:26 smithi116 bash[23740]: cluster 2023-07-10T17:08:25.626516+0000 mon.a (mon.0) 746 : cluster [DBG] mgrmap e21: x(active, since 2m), standbys: y 2023-07-10T17:08:26.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:26 smithi116 bash[23740]: cluster 2023-07-10T17:08:26.024611+0000 mgr.x (mgr.24857) 118 : cluster [DBG] pgmap v69: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:27.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:26 smithi192 bash[23730]: cluster 2023-07-10T17:08:25.626516+0000 mon.a (mon.0) 746 : cluster [DBG] mgrmap e21: x(active, since 2m), standbys: y 2023-07-10T17:08:27.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:26 smithi192 bash[23730]: cluster 2023-07-10T17:08:26.024611+0000 mgr.x (mgr.24857) 118 : cluster [DBG] pgmap v69: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:29.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:29 smithi192 bash[23730]: cluster 2023-07-10T17:08:28.025338+0000 mgr.x (mgr.24857) 119 : cluster [DBG] pgmap v70: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:29.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:29 smithi116 bash[23740]: cluster 2023-07-10T17:08:28.025338+0000 mgr.x (mgr.24857) 119 : cluster [DBG] pgmap v70: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:29.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:29 smithi116 bash[28351]: cluster 2023-07-10T17:08:28.025338+0000 mgr.x (mgr.24857) 119 : cluster [DBG] pgmap v70: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:31.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:31 smithi192 bash[23730]: cluster 2023-07-10T17:08:30.026115+0000 mgr.x (mgr.24857) 120 : cluster [DBG] pgmap v71: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:31.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:31 smithi116 bash[28351]: cluster 2023-07-10T17:08:30.026115+0000 mgr.x (mgr.24857) 120 : cluster [DBG] pgmap v71: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:31.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:31 smithi116 bash[23740]: cluster 2023-07-10T17:08:30.026115+0000 mgr.x (mgr.24857) 120 : cluster [DBG] pgmap v71: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:33.886 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:33 smithi192 bash[42233]: debug 2023-07-10T17:08:33.724+0000 7f7f59213700 -1 mgr handle_mgr_map I was active but no longer am 2023-07-10T17:08:33.887 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:33 smithi192 bash[42233]: ignoring --setuser ceph since I am not root 2023-07-10T17:08:33.887 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:33 smithi192 bash[42233]: ignoring --setgroup ceph since I am not root 2023-07-10T17:08:33.887 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:33 smithi192 bash[42233]: debug 2023-07-10T17:08:33.776+0000 7f3c55c6a700 1 -- 172.21.15.192:0/151771747 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559c983a91e0 con 0x559c99128800 2023-07-10T17:08:33.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:33 smithi192 bash[23730]: cluster 2023-07-10T17:08:32.027772+0000 mgr.x (mgr.24857) 121 : cluster [DBG] pgmap v72: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:33.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:33 smithi192 bash[23730]: audit 2023-07-10T17:08:32.683487+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:33.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:33 smithi192 bash[23730]: audit 2023-07-10T17:08:32.698094+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:33.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:33 smithi192 bash[23730]: audit 2023-07-10T17:08:32.702031+0000 mon.b (mon.2) 73 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:08:33.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:33 smithi192 bash[23730]: audit 2023-07-10T17:08:32.704004+0000 mon.b (mon.2) 74 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:08:33.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:33 smithi192 bash[23730]: audit 2023-07-10T17:08:32.715609+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:33.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:33 smithi192 bash[23730]: audit 2023-07-10T17:08:32.804836+0000 mon.b (mon.2) 75 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:08:33.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:33 smithi192 bash[23730]: audit 2023-07-10T17:08:32.814230+0000 mon.b (mon.2) 76 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-07-10T17:08:33.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:33 smithi192 bash[23730]: audit 2023-07-10T17:08:32.814631+0000 mon.a (mon.0) 750 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-07-10T17:08:33.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:33 smithi192 bash[23730]: cluster 2023-07-10T17:08:32.824638+0000 mon.a (mon.0) 751 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2023-07-10T17:08:33.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[23740]: cluster 2023-07-10T17:08:32.027772+0000 mgr.x (mgr.24857) 121 : cluster [DBG] pgmap v72: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:33.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[23740]: audit 2023-07-10T17:08:32.683487+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:33.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[23740]: audit 2023-07-10T17:08:32.698094+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:33.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[23740]: audit 2023-07-10T17:08:32.702031+0000 mon.b (mon.2) 73 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:08:33.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[23740]: audit 2023-07-10T17:08:32.704004+0000 mon.b (mon.2) 74 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:08:33.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[23740]: audit 2023-07-10T17:08:32.715609+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:33.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[23740]: audit 2023-07-10T17:08:32.804836+0000 mon.b (mon.2) 75 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:08:33.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[23740]: audit 2023-07-10T17:08:32.814230+0000 mon.b (mon.2) 76 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-07-10T17:08:33.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[23740]: audit 2023-07-10T17:08:32.814631+0000 mon.a (mon.0) 750 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-07-10T17:08:33.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[23740]: cluster 2023-07-10T17:08:32.824638+0000 mon.a (mon.0) 751 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2023-07-10T17:08:33.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[28351]: cluster 2023-07-10T17:08:32.027772+0000 mgr.x (mgr.24857) 121 : cluster [DBG] pgmap v72: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:33.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[28351]: audit 2023-07-10T17:08:32.683487+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:33.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[28351]: audit 2023-07-10T17:08:32.698094+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:33.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[28351]: audit 2023-07-10T17:08:32.702031+0000 mon.b (mon.2) 73 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:08:33.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[28351]: audit 2023-07-10T17:08:32.704004+0000 mon.b (mon.2) 74 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:08:33.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[28351]: audit 2023-07-10T17:08:32.715609+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.24857 ' entity='mgr.x' 2023-07-10T17:08:33.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[28351]: audit 2023-07-10T17:08:32.804836+0000 mon.b (mon.2) 75 : audit [DBG] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:08:33.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[28351]: audit 2023-07-10T17:08:32.814230+0000 mon.b (mon.2) 76 : audit [INF] from='mgr.24857 172.21.15.192:0/3476115782' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-07-10T17:08:33.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[28351]: audit 2023-07-10T17:08:32.814631+0000 mon.a (mon.0) 750 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-07-10T17:08:33.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:33 smithi116 bash[28351]: cluster 2023-07-10T17:08:32.824638+0000 mon.a (mon.0) 751 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2023-07-10T17:08:34.386 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:33 smithi192 bash[42233]: debug 2023-07-10T17:08:33.952+0000 7f3c5f99a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-10T17:08:34.856 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: cephadm 2023-07-10T17:08:32.705714+0000 mgr.x (mgr.24857) 122 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:08:34.856 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: cluster 2023-07-10T17:08:32.707084+0000 mgr.x (mgr.24857) 123 : cluster [DBG] pgmap v73: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:34.857 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: cluster 2023-07-10T17:08:32.707814+0000 mgr.x (mgr.24857) 124 : cluster [DBG] pgmap v74: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:34.857 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: cephadm 2023-07-10T17:08:32.810042+0000 mgr.x (mgr.24857) 125 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:08:34.857 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: cephadm 2023-07-10T17:08:32.810666+0000 mgr.x (mgr.24857) 126 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:08:34.858 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: cephadm 2023-07-10T17:08:32.813767+0000 mgr.x (mgr.24857) 127 : cephadm [INF] Failing over to other MGR 2023-07-10T17:08:34.858 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: cluster 2023-07-10T17:08:33.693946+0000 mon.a (mon.0) 752 : cluster [WRN] Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:08:34.858 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.725080+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2023-07-10T17:08:34.858 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: cluster 2023-07-10T17:08:33.725236+0000 mon.a (mon.0) 754 : cluster [DBG] mgrmap e22: y(active, starting, since 0.908991s) 2023-07-10T17:08:34.859 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.727605+0000 mon.c (mon.1) 52 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:08:34.859 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.727999+0000 mon.c (mon.1) 53 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:08:34.859 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.728446+0000 mon.c (mon.1) 54 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:08:34.860 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.729465+0000 mon.c (mon.1) 55 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-10T17:08:34.860 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.729866+0000 mon.c (mon.1) 56 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T17:08:34.860 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.730312+0000 mon.c (mon.1) 57 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T17:08:34.860 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.730760+0000 mon.c (mon.1) 58 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T17:08:34.860 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.731325+0000 mon.c (mon.1) 59 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T17:08:34.860 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.731753+0000 mon.c (mon.1) 60 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T17:08:34.861 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.732262+0000 mon.c (mon.1) 61 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T17:08:34.861 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.732723+0000 mon.c (mon.1) 62 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T17:08:34.861 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.733129+0000 mon.c (mon.1) 63 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T17:08:34.861 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.733512+0000 mon.c (mon.1) 64 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-10T17:08:34.861 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.733788+0000 mon.c (mon.1) 65 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-10T17:08:34.861 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.734358+0000 mon.c (mon.1) 66 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-10T17:08:34.861 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: cluster 2023-07-10T17:08:33.762390+0000 mon.a (mon.0) 755 : cluster [INF] Manager daemon y is now available 2023-07-10T17:08:34.862 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: cephadm 2023-07-10T17:08:33.786136+0000 mgr.y (mgr.15252) 1 : cephadm [INF] inventory: adjusted host smithi192 addr 'smithi192' -> '172.21.15.192' 2023-07-10T17:08:34.862 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.792741+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:34.862 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.812539+0000 mon.c (mon.1) 67 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:08:34.862 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.857524+0000 mon.c (mon.1) 68 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:08:34.862 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.858239+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:08:34.862 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.875459+0000 mon.c (mon.1) 69 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:08:34.862 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[23730]: audit 2023-07-10T17:08:33.875959+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:08:34.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: cephadm 2023-07-10T17:08:32.705714+0000 mgr.x (mgr.24857) 122 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:08:34.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: cluster 2023-07-10T17:08:32.707084+0000 mgr.x (mgr.24857) 123 : cluster [DBG] pgmap v73: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:34.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: cluster 2023-07-10T17:08:32.707814+0000 mgr.x (mgr.24857) 124 : cluster [DBG] pgmap v74: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:34.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: cephadm 2023-07-10T17:08:32.810042+0000 mgr.x (mgr.24857) 125 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:08:34.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: cephadm 2023-07-10T17:08:32.810666+0000 mgr.x (mgr.24857) 126 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:08:34.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: cephadm 2023-07-10T17:08:32.813767+0000 mgr.x (mgr.24857) 127 : cephadm [INF] Failing over to other MGR 2023-07-10T17:08:34.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: cluster 2023-07-10T17:08:33.693946+0000 mon.a (mon.0) 752 : cluster [WRN] Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:08:34.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.725080+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2023-07-10T17:08:34.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: cluster 2023-07-10T17:08:33.725236+0000 mon.a (mon.0) 754 : cluster [DBG] mgrmap e22: y(active, starting, since 0.908991s) 2023-07-10T17:08:34.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.727605+0000 mon.c (mon.1) 52 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:08:34.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.727999+0000 mon.c (mon.1) 53 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:08:34.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.728446+0000 mon.c (mon.1) 54 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:08:34.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.729465+0000 mon.c (mon.1) 55 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-10T17:08:34.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.729866+0000 mon.c (mon.1) 56 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T17:08:34.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.730312+0000 mon.c (mon.1) 57 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T17:08:34.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.730760+0000 mon.c (mon.1) 58 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T17:08:34.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.731325+0000 mon.c (mon.1) 59 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T17:08:34.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.731753+0000 mon.c (mon.1) 60 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T17:08:34.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.732262+0000 mon.c (mon.1) 61 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T17:08:34.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.732723+0000 mon.c (mon.1) 62 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T17:08:34.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.733129+0000 mon.c (mon.1) 63 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T17:08:34.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.733512+0000 mon.c (mon.1) 64 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-10T17:08:34.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.733788+0000 mon.c (mon.1) 65 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-10T17:08:34.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.734358+0000 mon.c (mon.1) 66 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-10T17:08:34.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: cluster 2023-07-10T17:08:33.762390+0000 mon.a (mon.0) 755 : cluster [INF] Manager daemon y is now available 2023-07-10T17:08:34.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: cephadm 2023-07-10T17:08:33.786136+0000 mgr.y (mgr.15252) 1 : cephadm [INF] inventory: adjusted host smithi192 addr 'smithi192' -> '172.21.15.192' 2023-07-10T17:08:34.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.792741+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:34.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.812539+0000 mon.c (mon.1) 67 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:08:34.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.857524+0000 mon.c (mon.1) 68 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:08:34.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.858239+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:08:34.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.875459+0000 mon.c (mon.1) 69 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:08:34.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[28351]: audit 2023-07-10T17:08:33.875959+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:08:34.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: cephadm 2023-07-10T17:08:32.705714+0000 mgr.x (mgr.24857) 122 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:08:34.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: cluster 2023-07-10T17:08:32.707084+0000 mgr.x (mgr.24857) 123 : cluster [DBG] pgmap v73: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:34.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: cluster 2023-07-10T17:08:32.707814+0000 mgr.x (mgr.24857) 124 : cluster [DBG] pgmap v74: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:34.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: cephadm 2023-07-10T17:08:32.810042+0000 mgr.x (mgr.24857) 125 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:08:34.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: cephadm 2023-07-10T17:08:32.810666+0000 mgr.x (mgr.24857) 126 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:08:34.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: cephadm 2023-07-10T17:08:32.813767+0000 mgr.x (mgr.24857) 127 : cephadm [INF] Failing over to other MGR 2023-07-10T17:08:34.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: cluster 2023-07-10T17:08:33.693946+0000 mon.a (mon.0) 752 : cluster [WRN] Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:08:34.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.725080+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.24857 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2023-07-10T17:08:34.900 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: cluster 2023-07-10T17:08:33.725236+0000 mon.a (mon.0) 754 : cluster [DBG] mgrmap e22: y(active, starting, since 0.908991s) 2023-07-10T17:08:34.900 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.727605+0000 mon.c (mon.1) 52 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:08:34.900 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.727999+0000 mon.c (mon.1) 53 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:08:34.900 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.728446+0000 mon.c (mon.1) 54 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:08:34.900 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.729465+0000 mon.c (mon.1) 55 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-10T17:08:34.900 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.729866+0000 mon.c (mon.1) 56 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T17:08:34.900 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.730312+0000 mon.c (mon.1) 57 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T17:08:34.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.730760+0000 mon.c (mon.1) 58 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T17:08:34.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.731325+0000 mon.c (mon.1) 59 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T17:08:34.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.731753+0000 mon.c (mon.1) 60 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T17:08:34.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.732262+0000 mon.c (mon.1) 61 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T17:08:34.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.732723+0000 mon.c (mon.1) 62 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T17:08:34.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.733129+0000 mon.c (mon.1) 63 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T17:08:34.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.733512+0000 mon.c (mon.1) 64 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-10T17:08:34.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.733788+0000 mon.c (mon.1) 65 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-10T17:08:34.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.734358+0000 mon.c (mon.1) 66 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-10T17:08:34.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: cluster 2023-07-10T17:08:33.762390+0000 mon.a (mon.0) 755 : cluster [INF] Manager daemon y is now available 2023-07-10T17:08:34.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: cephadm 2023-07-10T17:08:33.786136+0000 mgr.y (mgr.15252) 1 : cephadm [INF] inventory: adjusted host smithi192 addr 'smithi192' -> '172.21.15.192' 2023-07-10T17:08:34.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.792741+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:34.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.812539+0000 mon.c (mon.1) 67 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:08:34.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.857524+0000 mon.c (mon.1) 68 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:08:34.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.858239+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:08:34.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.875459+0000 mon.c (mon.1) 69 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:08:34.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:34 smithi116 bash[23740]: audit 2023-07-10T17:08:33.875959+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:08:35.136 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[42233]: debug 2023-07-10T17:08:34.848+0000 7f3c5f99a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-10T17:08:35.137 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:34 smithi192 bash[42233]: debug 2023-07-10T17:08:34.948+0000 7f3c5f99a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-10T17:08:35.137 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:35 smithi192 bash[42233]: debug 2023-07-10T17:08:35.048+0000 7f3c5f99a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-10T17:08:35.636 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:35 smithi192 bash[42233]: debug 2023-07-10T17:08:35.300+0000 7f3c5f99a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-10T17:08:36.028 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:35 smithi192 bash[42233]: debug 2023-07-10T17:08:35.652+0000 7f3c5f99a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-10T17:08:36.028 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:35 smithi192 bash[42233]: debug 2023-07-10T17:08:35.812+0000 7f3c5f99a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-10T17:08:36.029 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:35 smithi192 bash[23730]: cluster 2023-07-10T17:08:34.732366+0000 mon.a (mon.0) 759 : cluster [DBG] mgrmap e23: y(active, since 1.91612s) 2023-07-10T17:08:36.029 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:35 smithi192 bash[23730]: cluster 2023-07-10T17:08:34.762948+0000 mgr.y (mgr.15252) 2 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:36.029 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:35 smithi192 bash[23730]: cephadm 2023-07-10T17:08:34.964152+0000 mgr.y (mgr.15252) 3 : cephadm [INF] [10/Jul/2023:17:08:34] ENGINE Bus STARTING 2023-07-10T17:08:36.029 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:35 smithi192 bash[23730]: cephadm 2023-07-10T17:08:35.066231+0000 mgr.y (mgr.15252) 4 : cephadm [INF] [10/Jul/2023:17:08:35] ENGINE Serving on http://172.21.15.116:8765 2023-07-10T17:08:36.029 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:35 smithi192 bash[23730]: cephadm 2023-07-10T17:08:35.193673+0000 mgr.y (mgr.15252) 5 : cephadm [INF] [10/Jul/2023:17:08:35] ENGINE Serving on https://172.21.15.116:7150 2023-07-10T17:08:36.030 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:35 smithi192 bash[23730]: cephadm 2023-07-10T17:08:35.194073+0000 mgr.y (mgr.15252) 6 : cephadm [INF] [10/Jul/2023:17:08:35] ENGINE Bus STARTED 2023-07-10T17:08:36.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:35 smithi116 bash[28351]: cluster 2023-07-10T17:08:34.732366+0000 mon.a (mon.0) 759 : cluster [DBG] mgrmap e23: y(active, since 1.91612s) 2023-07-10T17:08:36.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:35 smithi116 bash[28351]: cluster 2023-07-10T17:08:34.762948+0000 mgr.y (mgr.15252) 2 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:36.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:35 smithi116 bash[28351]: cephadm 2023-07-10T17:08:34.964152+0000 mgr.y (mgr.15252) 3 : cephadm [INF] [10/Jul/2023:17:08:34] ENGINE Bus STARTING 2023-07-10T17:08:36.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:35 smithi116 bash[28351]: cephadm 2023-07-10T17:08:35.066231+0000 mgr.y (mgr.15252) 4 : cephadm [INF] [10/Jul/2023:17:08:35] ENGINE Serving on http://172.21.15.116:8765 2023-07-10T17:08:36.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:35 smithi116 bash[28351]: cephadm 2023-07-10T17:08:35.193673+0000 mgr.y (mgr.15252) 5 : cephadm [INF] [10/Jul/2023:17:08:35] ENGINE Serving on https://172.21.15.116:7150 2023-07-10T17:08:36.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:35 smithi116 bash[28351]: cephadm 2023-07-10T17:08:35.194073+0000 mgr.y (mgr.15252) 6 : cephadm [INF] [10/Jul/2023:17:08:35] ENGINE Bus STARTED 2023-07-10T17:08:36.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:35 smithi116 bash[23740]: cluster 2023-07-10T17:08:34.732366+0000 mon.a (mon.0) 759 : cluster [DBG] mgrmap e23: y(active, since 1.91612s) 2023-07-10T17:08:36.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:35 smithi116 bash[23740]: cluster 2023-07-10T17:08:34.762948+0000 mgr.y (mgr.15252) 2 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:36.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:35 smithi116 bash[23740]: cephadm 2023-07-10T17:08:34.964152+0000 mgr.y (mgr.15252) 3 : cephadm [INF] [10/Jul/2023:17:08:34] ENGINE Bus STARTING 2023-07-10T17:08:36.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:35 smithi116 bash[23740]: cephadm 2023-07-10T17:08:35.066231+0000 mgr.y (mgr.15252) 4 : cephadm [INF] [10/Jul/2023:17:08:35] ENGINE Serving on http://172.21.15.116:8765 2023-07-10T17:08:36.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:35 smithi116 bash[23740]: cephadm 2023-07-10T17:08:35.193673+0000 mgr.y (mgr.15252) 5 : cephadm [INF] [10/Jul/2023:17:08:35] ENGINE Serving on https://172.21.15.116:7150 2023-07-10T17:08:36.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:35 smithi116 bash[23740]: cephadm 2023-07-10T17:08:35.194073+0000 mgr.y (mgr.15252) 6 : cephadm [INF] [10/Jul/2023:17:08:35] ENGINE Bus STARTED 2023-07-10T17:08:36.386 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:36 smithi192 bash[42233]: debug 2023-07-10T17:08:36.024+0000 7f3c5f99a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-10T17:08:37.017 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:36 smithi192 bash[23730]: cluster 2023-07-10T17:08:35.723093+0000 mon.a (mon.0) 760 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2023-07-10T17:08:37.018 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:36 smithi192 bash[23730]: cluster 2023-07-10T17:08:35.723217+0000 mon.a (mon.0) 761 : cluster [INF] Cluster is now healthy 2023-07-10T17:08:37.018 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:36 smithi192 bash[23730]: cluster 2023-07-10T17:08:35.731758+0000 mgr.y (mgr.15252) 7 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:37.018 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:36 smithi192 bash[23730]: cluster 2023-07-10T17:08:35.744918+0000 mon.a (mon.0) 762 : cluster [DBG] mgrmap e24: y(active, since 2s) 2023-07-10T17:08:37.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:36 smithi116 bash[23740]: cluster 2023-07-10T17:08:35.723093+0000 mon.a (mon.0) 760 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2023-07-10T17:08:37.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:36 smithi116 bash[23740]: cluster 2023-07-10T17:08:35.723217+0000 mon.a (mon.0) 761 : cluster [INF] Cluster is now healthy 2023-07-10T17:08:37.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:36 smithi116 bash[23740]: cluster 2023-07-10T17:08:35.731758+0000 mgr.y (mgr.15252) 7 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:37.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:36 smithi116 bash[23740]: cluster 2023-07-10T17:08:35.744918+0000 mon.a (mon.0) 762 : cluster [DBG] mgrmap e24: y(active, since 2s) 2023-07-10T17:08:37.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:36 smithi116 bash[28351]: cluster 2023-07-10T17:08:35.723093+0000 mon.a (mon.0) 760 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2023-07-10T17:08:37.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:36 smithi116 bash[28351]: cluster 2023-07-10T17:08:35.723217+0000 mon.a (mon.0) 761 : cluster [INF] Cluster is now healthy 2023-07-10T17:08:37.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:36 smithi116 bash[28351]: cluster 2023-07-10T17:08:35.731758+0000 mgr.y (mgr.15252) 7 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:37.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:36 smithi116 bash[28351]: cluster 2023-07-10T17:08:35.744918+0000 mon.a (mon.0) 762 : cluster [DBG] mgrmap e24: y(active, since 2s) 2023-07-10T17:08:37.386 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:37 smithi192 bash[42233]: debug 2023-07-10T17:08:37.012+0000 7f3c5f99a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-10T17:08:37.387 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:37 smithi192 bash[42233]: debug 2023-07-10T17:08:37.120+0000 7f3c5f99a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-10T17:08:37.387 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:37 smithi192 bash[42233]: debug 2023-07-10T17:08:37.236+0000 7f3c5f99a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-10T17:08:38.136 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:37 smithi192 bash[42233]: debug 2023-07-10T17:08:37.708+0000 7f3c5f99a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-10T17:08:38.137 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:37 smithi192 bash[42233]: debug 2023-07-10T17:08:37.928+0000 7f3c5f99a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-10T17:08:38.636 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:38 smithi192 bash[42233]: debug 2023-07-10T17:08:38.296+0000 7f3c5f99a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-10T17:08:39.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:38 smithi192 bash[23730]: cluster 2023-07-10T17:08:37.732718+0000 mgr.y (mgr.15252) 8 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:39.137 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:38 smithi192 bash[42233]: debug 2023-07-10T17:08:38.784+0000 7f3c5f99a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-10T17:08:39.142 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:38 smithi116 bash[23740]: cluster 2023-07-10T17:08:37.732718+0000 mgr.y (mgr.15252) 8 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:39.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:38 smithi116 bash[28351]: cluster 2023-07-10T17:08:37.732718+0000 mgr.y (mgr.15252) 8 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:39.636 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:39 smithi192 bash[42233]: debug 2023-07-10T17:08:39.332+0000 7f3c5f99a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-10T17:08:39.637 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:39 smithi192 bash[42233]: debug 2023-07-10T17:08:39.448+0000 7f3c5f99a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-10T17:08:40.113 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:39 smithi192 bash[42233]: debug 2023-07-10T17:08:39.832+0000 7f3c5f99a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-10T17:08:40.114 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:39 smithi192 bash[42233]: debug 2023-07-10T17:08:39.952+0000 7f3c5f99a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-10T17:08:40.366 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:40 smithi192 bash[42233]: debug 2023-07-10T17:08:40.108+0000 7f3c5f99a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-10T17:08:40.367 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:40 smithi192 bash[42233]: debug 2023-07-10T17:08:40.228+0000 7f3c5f99a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-10T17:08:40.636 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:40 smithi192 bash[42233]: debug 2023-07-10T17:08:40.360+0000 7f3c5f99a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-10T17:08:40.637 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:40 smithi192 bash[42233]: debug 2023-07-10T17:08:40.568+0000 7f3c5f99a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-10T17:08:41.136 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:40 smithi192 bash[42233]: debug 2023-07-10T17:08:40.700+0000 7f3c5f99a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-10T17:08:41.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:40 smithi192 bash[23730]: cluster 2023-07-10T17:08:39.733623+0000 mgr.y (mgr.15252) 9 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:41.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:40 smithi192 bash[23730]: audit 2023-07-10T17:08:40.707514+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:41.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:40 smithi192 bash[23730]: audit 2023-07-10T17:08:40.715859+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:41.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:40 smithi116 bash[23740]: cluster 2023-07-10T17:08:39.733623+0000 mgr.y (mgr.15252) 9 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:41.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:40 smithi116 bash[23740]: audit 2023-07-10T17:08:40.707514+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:41.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:40 smithi116 bash[23740]: audit 2023-07-10T17:08:40.715859+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:41.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:40 smithi116 bash[28351]: cluster 2023-07-10T17:08:39.733623+0000 mgr.y (mgr.15252) 9 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:41.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:40 smithi116 bash[28351]: audit 2023-07-10T17:08:40.707514+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:41.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:40 smithi116 bash[28351]: audit 2023-07-10T17:08:40.715859+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.386 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:08:41 smithi192 bash[42233]: debug 2023-07-10T17:08:41.940+0000 7f3c5f99a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-10T17:08:42.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:41 smithi192 bash[23730]: audit 2023-07-10T17:08:40.905339+0000 mon.a (mon.0) 765 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:41 smithi192 bash[23730]: audit 2023-07-10T17:08:40.914426+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:41 smithi192 bash[23730]: audit 2023-07-10T17:08:41.469517+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:41 smithi192 bash[23730]: audit 2023-07-10T17:08:41.481855+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:41 smithi192 bash[23730]: audit 2023-07-10T17:08:41.720026+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:41 smithi192 bash[23730]: audit 2023-07-10T17:08:41.733336+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:41 smithi116 bash[28351]: audit 2023-07-10T17:08:40.905339+0000 mon.a (mon.0) 765 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:41 smithi116 bash[28351]: audit 2023-07-10T17:08:40.914426+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:41 smithi116 bash[28351]: audit 2023-07-10T17:08:41.469517+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:41 smithi116 bash[28351]: audit 2023-07-10T17:08:41.481855+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:41 smithi116 bash[28351]: audit 2023-07-10T17:08:41.720026+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:41 smithi116 bash[28351]: audit 2023-07-10T17:08:41.733336+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:41 smithi116 bash[23740]: audit 2023-07-10T17:08:40.905339+0000 mon.a (mon.0) 765 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:41 smithi116 bash[23740]: audit 2023-07-10T17:08:40.914426+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:41 smithi116 bash[23740]: audit 2023-07-10T17:08:41.469517+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:41 smithi116 bash[23740]: audit 2023-07-10T17:08:41.481855+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:41 smithi116 bash[23740]: audit 2023-07-10T17:08:41.720026+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:42.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:41 smithi116 bash[23740]: audit 2023-07-10T17:08:41.733336+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:43.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:42 smithi192 bash[23730]: cluster 2023-07-10T17:08:41.734458+0000 mgr.y (mgr.15252) 10 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:43.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:42 smithi192 bash[23730]: audit 2023-07-10T17:08:41.945148+0000 mon.b (mon.2) 77 : audit [DBG] from='mgr.? 172.21.15.192:0/2230669237' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-10T17:08:43.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:42 smithi192 bash[23730]: cluster 2023-07-10T17:08:41.945704+0000 mon.a (mon.0) 771 : cluster [DBG] Standby manager daemon x started 2023-07-10T17:08:43.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:42 smithi192 bash[23730]: audit 2023-07-10T17:08:41.950391+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.? 172.21.15.192:0/2230669237' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:08:43.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:42 smithi192 bash[23730]: audit 2023-07-10T17:08:41.951506+0000 mon.b (mon.2) 79 : audit [DBG] from='mgr.? 172.21.15.192:0/2230669237' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-10T17:08:43.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:42 smithi192 bash[23730]: audit 2023-07-10T17:08:41.952023+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.? 172.21.15.192:0/2230669237' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:08:43.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:42 smithi116 bash[28351]: cluster 2023-07-10T17:08:41.734458+0000 mgr.y (mgr.15252) 10 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:43.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:42 smithi116 bash[28351]: audit 2023-07-10T17:08:41.945148+0000 mon.b (mon.2) 77 : audit [DBG] from='mgr.? 172.21.15.192:0/2230669237' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-10T17:08:43.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:42 smithi116 bash[28351]: cluster 2023-07-10T17:08:41.945704+0000 mon.a (mon.0) 771 : cluster [DBG] Standby manager daemon x started 2023-07-10T17:08:43.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:42 smithi116 bash[28351]: audit 2023-07-10T17:08:41.950391+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.? 172.21.15.192:0/2230669237' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:08:43.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:42 smithi116 bash[28351]: audit 2023-07-10T17:08:41.951506+0000 mon.b (mon.2) 79 : audit [DBG] from='mgr.? 172.21.15.192:0/2230669237' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-10T17:08:43.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:42 smithi116 bash[28351]: audit 2023-07-10T17:08:41.952023+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.? 172.21.15.192:0/2230669237' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:08:43.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:42 smithi116 bash[23740]: cluster 2023-07-10T17:08:41.734458+0000 mgr.y (mgr.15252) 10 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:43.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:42 smithi116 bash[23740]: audit 2023-07-10T17:08:41.945148+0000 mon.b (mon.2) 77 : audit [DBG] from='mgr.? 172.21.15.192:0/2230669237' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-10T17:08:43.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:42 smithi116 bash[23740]: cluster 2023-07-10T17:08:41.945704+0000 mon.a (mon.0) 771 : cluster [DBG] Standby manager daemon x started 2023-07-10T17:08:43.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:42 smithi116 bash[23740]: audit 2023-07-10T17:08:41.950391+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.? 172.21.15.192:0/2230669237' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:08:43.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:42 smithi116 bash[23740]: audit 2023-07-10T17:08:41.951506+0000 mon.b (mon.2) 79 : audit [DBG] from='mgr.? 172.21.15.192:0/2230669237' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-10T17:08:43.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:42 smithi116 bash[23740]: audit 2023-07-10T17:08:41.952023+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.? 172.21.15.192:0/2230669237' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:08:44.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:43 smithi192 bash[23730]: cluster 2023-07-10T17:08:42.923290+0000 mon.a (mon.0) 772 : cluster [DBG] mgrmap e25: y(active, since 10s), standbys: x 2023-07-10T17:08:44.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:43 smithi192 bash[23730]: audit 2023-07-10T17:08:42.929323+0000 mon.c (mon.1) 70 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-10T17:08:44.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:43 smithi116 bash[28351]: cluster 2023-07-10T17:08:42.923290+0000 mon.a (mon.0) 772 : cluster [DBG] mgrmap e25: y(active, since 10s), standbys: x 2023-07-10T17:08:44.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:43 smithi116 bash[28351]: audit 2023-07-10T17:08:42.929323+0000 mon.c (mon.1) 70 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-10T17:08:44.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:43 smithi116 bash[23740]: cluster 2023-07-10T17:08:42.923290+0000 mon.a (mon.0) 772 : cluster [DBG] mgrmap e25: y(active, since 10s), standbys: x 2023-07-10T17:08:44.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:43 smithi116 bash[23740]: audit 2023-07-10T17:08:42.929323+0000 mon.c (mon.1) 70 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-10T17:08:45.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:44 smithi192 bash[23730]: cluster 2023-07-10T17:08:43.735215+0000 mgr.y (mgr.15252) 11 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:45.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:44 smithi116 bash[23740]: cluster 2023-07-10T17:08:43.735215+0000 mgr.y (mgr.15252) 11 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:45.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:44 smithi116 bash[28351]: cluster 2023-07-10T17:08:43.735215+0000 mgr.y (mgr.15252) 11 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:47.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:46 smithi192 bash[23730]: cluster 2023-07-10T17:08:45.736861+0000 mgr.y (mgr.15252) 12 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:47.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:46 smithi116 bash[23740]: cluster 2023-07-10T17:08:45.736861+0000 mgr.y (mgr.15252) 12 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:47.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:46 smithi116 bash[28351]: cluster 2023-07-10T17:08:45.736861+0000 mgr.y (mgr.15252) 12 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:49.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:48 smithi116 bash[23740]: cluster 2023-07-10T17:08:47.737645+0000 mgr.y (mgr.15252) 13 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:49.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:48 smithi116 bash[28351]: cluster 2023-07-10T17:08:47.737645+0000 mgr.y (mgr.15252) 13 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:49.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:48 smithi192 bash[23730]: cluster 2023-07-10T17:08:47.737645+0000 mgr.y (mgr.15252) 13 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:51.219 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:50 smithi116 bash[23740]: cluster 2023-07-10T17:08:49.738577+0000 mgr.y (mgr.15252) 14 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:51.219 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:50 smithi116 bash[28351]: cluster 2023-07-10T17:08:49.738577+0000 mgr.y (mgr.15252) 14 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:51.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:50 smithi192 bash[23730]: cluster 2023-07-10T17:08:49.738577+0000 mgr.y (mgr.15252) 14 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:52.246 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:08:52.976 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:08:52.977 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (2m) 12s ago 8m 9.94M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:08:52.977 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (44s) 12s ago 8m 24.1M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:08:52.977 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 error 12s ago 5m - - 2023-07-10T17:08:52.977 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443 running (2m) 12s ago 12m 157M - 18.1.2-30-g8507faa7 17e8f0d79247 d22a6b7f47df 2023-07-10T17:08:52.977 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (37s) 12s ago 14m 453M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:08:52.977 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (14m) 12s ago 14m 61.7M 2048M 16.2.0 24ecd6d5f14c 33f26d450a67 2023-07-10T17:08:52.977 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (12m) 12s ago 12m 54.8M 2048M 16.2.0 24ecd6d5f14c 9ad4329e5488 2023-07-10T17:08:52.978 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (13m) 12s ago 13m 49.3M 2048M 16.2.0 24ecd6d5f14c 31e3ac12741b 2023-07-10T17:08:52.978 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (2m) 12s ago 8m 8987k - 0.18.1 e5a616e4b9cf 3bd058d91353 2023-07-10T17:08:52.978 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (101s) 12s ago 8m 3172k - 0.18.1 e5a616e4b9cf 63551615c679 2023-07-10T17:08:52.978 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (12m) 12s ago 12m 54.5M 4096M 16.2.0 24ecd6d5f14c 4fa26367c830 2023-07-10T17:08:52.978 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (11m) 12s ago 11m 53.0M 4096M 16.2.0 24ecd6d5f14c c157509b3fd4 2023-07-10T17:08:52.978 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (11m) 12s ago 11m 49.0M 4096M 16.2.0 24ecd6d5f14c f1fd77b3c454 2023-07-10T17:08:52.978 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (10m) 12s ago 10m 54.2M 4096M 16.2.0 24ecd6d5f14c 60949d479c11 2023-07-10T17:08:52.979 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (10m) 12s ago 10m 53.6M 4096M 16.2.0 24ecd6d5f14c 75fc0b36aab8 2023-07-10T17:08:52.979 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (10m) 12s ago 10m 51.6M 4096M 16.2.0 24ecd6d5f14c d673065333dd 2023-07-10T17:08:52.979 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (9m) 12s ago 9m 52.7M 4096M 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:08:52.979 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (9m) 12s ago 9m 55.5M 4096M 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:08:52.979 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 error 12s ago 8m - - 2023-07-10T17:08:52.979 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (6m) 12s ago 6m 53.5M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:08:52.979 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (6m) 12s ago 6m 53.6M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:08:52.979 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (6m) 12s ago 6m 53.9M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:08:52.980 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (6m) 12s ago 6m 55.4M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:08:53.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:52 smithi192 bash[23730]: cluster 2023-07-10T17:08:51.740244+0000 mgr.y (mgr.15252) 15 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:53.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:52 smithi192 bash[23730]: audit 2023-07-10T17:08:51.836421+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:53.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:52 smithi192 bash[23730]: audit 2023-07-10T17:08:51.848922+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:53.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:52 smithi192 bash[23730]: audit 2023-07-10T17:08:51.854306+0000 mon.c (mon.1) 71 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:08:53.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:52 smithi192 bash[23730]: audit 2023-07-10T17:08:51.855925+0000 mon.c (mon.1) 72 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:08:53.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:52 smithi192 bash[23730]: cephadm 2023-07-10T17:08:51.857316+0000 mgr.y (mgr.15252) 16 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:08:53.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:52 smithi192 bash[23730]: cluster 2023-07-10T17:08:51.858698+0000 mgr.y (mgr.15252) 17 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:53.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:52 smithi192 bash[23730]: audit 2023-07-10T17:08:51.866075+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:53.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:52 smithi192 bash[23730]: cluster 2023-07-10T17:08:51.940376+0000 mon.a (mon.0) 776 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:08:53.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:52 smithi192 bash[23730]: cephadm 2023-07-10T17:08:51.943607+0000 mgr.y (mgr.15252) 18 : cephadm [INF] Reconfiguring iscsi.foo.smithi116.fkthyl (dependencies changed)... 2023-07-10T17:08:53.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:52 smithi192 bash[23730]: audit 2023-07-10T17:08:51.948481+0000 mon.c (mon.1) 73 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:08:53.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:52 smithi192 bash[23730]: audit 2023-07-10T17:08:51.949329+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:08:53.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:52 smithi192 bash[23730]: audit 2023-07-10T17:08:51.964975+0000 mon.c (mon.1) 74 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:08:53.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:52 smithi192 bash[23730]: cephadm 2023-07-10T17:08:51.966478+0000 mgr.y (mgr.15252) 19 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:08:53.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:52 smithi192 bash[23730]: audit 2023-07-10T17:08:52.227880+0000 mgr.y (mgr.15252) 20 : audit [DBG] from='client.25237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:53.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[23740]: cluster 2023-07-10T17:08:51.740244+0000 mgr.y (mgr.15252) 15 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:53.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[23740]: audit 2023-07-10T17:08:51.836421+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:53.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[23740]: audit 2023-07-10T17:08:51.848922+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:53.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[23740]: audit 2023-07-10T17:08:51.854306+0000 mon.c (mon.1) 71 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:08:53.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[23740]: audit 2023-07-10T17:08:51.855925+0000 mon.c (mon.1) 72 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:08:53.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[23740]: cephadm 2023-07-10T17:08:51.857316+0000 mgr.y (mgr.15252) 16 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:08:53.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[23740]: cluster 2023-07-10T17:08:51.858698+0000 mgr.y (mgr.15252) 17 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:53.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[23740]: audit 2023-07-10T17:08:51.866075+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:53.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[23740]: cluster 2023-07-10T17:08:51.940376+0000 mon.a (mon.0) 776 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:08:53.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[23740]: cephadm 2023-07-10T17:08:51.943607+0000 mgr.y (mgr.15252) 18 : cephadm [INF] Reconfiguring iscsi.foo.smithi116.fkthyl (dependencies changed)... 2023-07-10T17:08:53.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[23740]: audit 2023-07-10T17:08:51.948481+0000 mon.c (mon.1) 73 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:08:53.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[23740]: audit 2023-07-10T17:08:51.949329+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:08:53.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[23740]: audit 2023-07-10T17:08:51.964975+0000 mon.c (mon.1) 74 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:08:53.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[23740]: cephadm 2023-07-10T17:08:51.966478+0000 mgr.y (mgr.15252) 19 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:08:53.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[23740]: audit 2023-07-10T17:08:52.227880+0000 mgr.y (mgr.15252) 20 : audit [DBG] from='client.25237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:53.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[28351]: cluster 2023-07-10T17:08:51.740244+0000 mgr.y (mgr.15252) 15 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:53.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[28351]: audit 2023-07-10T17:08:51.836421+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:53.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[28351]: audit 2023-07-10T17:08:51.848922+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:53.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[28351]: audit 2023-07-10T17:08:51.854306+0000 mon.c (mon.1) 71 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:08:53.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[28351]: audit 2023-07-10T17:08:51.855925+0000 mon.c (mon.1) 72 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:08:53.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[28351]: cephadm 2023-07-10T17:08:51.857316+0000 mgr.y (mgr.15252) 16 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:08:53.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[28351]: cluster 2023-07-10T17:08:51.858698+0000 mgr.y (mgr.15252) 17 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:53.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[28351]: audit 2023-07-10T17:08:51.866075+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:53.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[28351]: cluster 2023-07-10T17:08:51.940376+0000 mon.a (mon.0) 776 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:08:53.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[28351]: cephadm 2023-07-10T17:08:51.943607+0000 mgr.y (mgr.15252) 18 : cephadm [INF] Reconfiguring iscsi.foo.smithi116.fkthyl (dependencies changed)... 2023-07-10T17:08:53.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[28351]: audit 2023-07-10T17:08:51.948481+0000 mon.c (mon.1) 73 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:08:53.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[28351]: audit 2023-07-10T17:08:51.949329+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:08:53.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[28351]: audit 2023-07-10T17:08:51.964975+0000 mon.c (mon.1) 74 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:08:53.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[28351]: cephadm 2023-07-10T17:08:51.966478+0000 mgr.y (mgr.15252) 19 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:08:53.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:52 smithi116 bash[28351]: audit 2023-07-10T17:08:52.227880+0000 mgr.y (mgr.15252) 20 : audit [DBG] from='client.25237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:53.425 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:08:53.425 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:08:53.425 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2023-07-10T17:08:53.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:08:53.425 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:08:53.426 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:08:53.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:08:53.426 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:08:53.426 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-10T17:08:53.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:08:53.426 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": {}, 2023-07-10T17:08:53.426 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:08:53.426 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:08:53.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:08:53.427 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:08:53.427 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 15, 2023-07-10T17:08:53.427 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:08:53.427 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:08:53.427 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:08:53.771 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:08:53.771 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:08:53.771 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:08:53.771 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:08:53.771 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:08:53.771 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2023-07-10T17:08:53.772 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:08:53.772 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "2/23 daemons upgraded", 2023-07-10T17:08:53.772 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "", 2023-07-10T17:08:53.772 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:08:53.772 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:08:54.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:53 smithi192 bash[23730]: audit 2023-07-10T17:08:52.588507+0000 mgr.y (mgr.15252) 21 : audit [DBG] from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:54.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:53 smithi192 bash[23730]: audit 2023-07-10T17:08:52.948936+0000 mgr.y (mgr.15252) 22 : audit [DBG] from='client.25249 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:54.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:53 smithi192 bash[23730]: audit 2023-07-10T17:08:53.421417+0000 mon.a (mon.0) 778 : audit [DBG] from='client.? 172.21.15.116:0/1134791212' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:08:54.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:53 smithi116 bash[28351]: audit 2023-07-10T17:08:52.588507+0000 mgr.y (mgr.15252) 21 : audit [DBG] from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:54.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:53 smithi116 bash[28351]: audit 2023-07-10T17:08:52.948936+0000 mgr.y (mgr.15252) 22 : audit [DBG] from='client.25249 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:54.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:53 smithi116 bash[28351]: audit 2023-07-10T17:08:53.421417+0000 mon.a (mon.0) 778 : audit [DBG] from='client.? 172.21.15.116:0/1134791212' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:08:54.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:53 smithi116 bash[23740]: audit 2023-07-10T17:08:52.588507+0000 mgr.y (mgr.15252) 21 : audit [DBG] from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:54.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:53 smithi116 bash[23740]: audit 2023-07-10T17:08:52.948936+0000 mgr.y (mgr.15252) 22 : audit [DBG] from='client.25249 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:54.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:53 smithi116 bash[23740]: audit 2023-07-10T17:08:53.421417+0000 mon.a (mon.0) 778 : audit [DBG] from='client.? 172.21.15.116:0/1134791212' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:08:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: daemon prometheus.a on smithi192 is in error state 2023-07-10T17:08:55.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:54 smithi192 bash[23730]: audit 2023-07-10T17:08:53.767153+0000 mgr.y (mgr.15252) 23 : audit [DBG] from='client.25261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:55.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:54 smithi192 bash[23730]: cluster 2023-07-10T17:08:53.859478+0000 mgr.y (mgr.15252) 24 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:55.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:54 smithi192 bash[23730]: audit 2023-07-10T17:08:54.182974+0000 mon.a (mon.0) 779 : audit [DBG] from='client.? 172.21.15.116:0/3310881407' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:08:55.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:54 smithi192 bash[23730]: audit 2023-07-10T17:08:54.241867+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:55.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:54 smithi192 bash[23730]: audit 2023-07-10T17:08:54.252383+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:55.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:54 smithi192 bash[23730]: cephadm 2023-07-10T17:08:54.286766+0000 mgr.y (mgr.15252) 25 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-10T17:08:55.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:54 smithi116 bash[23740]: audit 2023-07-10T17:08:53.767153+0000 mgr.y (mgr.15252) 23 : audit [DBG] from='client.25261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:55.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:54 smithi116 bash[23740]: cluster 2023-07-10T17:08:53.859478+0000 mgr.y (mgr.15252) 24 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:55.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:54 smithi116 bash[23740]: audit 2023-07-10T17:08:54.182974+0000 mon.a (mon.0) 779 : audit [DBG] from='client.? 172.21.15.116:0/3310881407' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:08:55.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:54 smithi116 bash[23740]: audit 2023-07-10T17:08:54.241867+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:55.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:54 smithi116 bash[23740]: audit 2023-07-10T17:08:54.252383+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:55.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:54 smithi116 bash[23740]: cephadm 2023-07-10T17:08:54.286766+0000 mgr.y (mgr.15252) 25 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-10T17:08:55.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:54 smithi116 bash[28351]: audit 2023-07-10T17:08:53.767153+0000 mgr.y (mgr.15252) 23 : audit [DBG] from='client.25261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:08:55.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:54 smithi116 bash[28351]: cluster 2023-07-10T17:08:53.859478+0000 mgr.y (mgr.15252) 24 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:55.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:54 smithi116 bash[28351]: audit 2023-07-10T17:08:54.182974+0000 mon.a (mon.0) 779 : audit [DBG] from='client.? 172.21.15.116:0/3310881407' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:08:55.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:54 smithi116 bash[28351]: audit 2023-07-10T17:08:54.241867+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:55.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:54 smithi116 bash[28351]: audit 2023-07-10T17:08:54.252383+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:08:55.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:54 smithi116 bash[28351]: cephadm 2023-07-10T17:08:54.286766+0000 mgr.y (mgr.15252) 25 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-07-10T17:08:55.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:55 smithi116 bash[28351]: cephadm 2023-07-10T17:08:54.529934+0000 mgr.y (mgr.15252) 26 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi192 2023-07-10T17:08:55.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:55 smithi116 bash[23740]: cephadm 2023-07-10T17:08:54.529934+0000 mgr.y (mgr.15252) 26 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi192 2023-07-10T17:08:56.125 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:55 smithi192 bash[23730]: cephadm 2023-07-10T17:08:54.529934+0000 mgr.y (mgr.15252) 26 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi192 2023-07-10T17:08:57.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:56 smithi192 bash[23730]: cluster 2023-07-10T17:08:55.860546+0000 mgr.y (mgr.15252) 27 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:57.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:56 smithi192 bash[23730]: audit 2023-07-10T17:08:56.313524+0000 mon.a (mon.0) 782 : audit [DBG] from='client.? 172.21.15.116:0/831612487' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:08:57.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:56 smithi192 bash[23730]: audit 2023-07-10T17:08:56.685996+0000 mon.c (mon.1) 75 : audit [INF] from='client.? 172.21.15.116:0/2387753186' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1382514600"}]: access denied 2023-07-10T17:08:57.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:56 smithi116 bash[23740]: cluster 2023-07-10T17:08:55.860546+0000 mgr.y (mgr.15252) 27 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:57.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:56 smithi116 bash[23740]: audit 2023-07-10T17:08:56.313524+0000 mon.a (mon.0) 782 : audit [DBG] from='client.? 172.21.15.116:0/831612487' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:08:57.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:56 smithi116 bash[23740]: audit 2023-07-10T17:08:56.685996+0000 mon.c (mon.1) 75 : audit [INF] from='client.? 172.21.15.116:0/2387753186' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1382514600"}]: access denied 2023-07-10T17:08:57.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:56 smithi116 bash[28351]: cluster 2023-07-10T17:08:55.860546+0000 mgr.y (mgr.15252) 27 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:57.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:56 smithi116 bash[28351]: audit 2023-07-10T17:08:56.313524+0000 mon.a (mon.0) 782 : audit [DBG] from='client.? 172.21.15.116:0/831612487' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:08:57.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:56 smithi116 bash[28351]: audit 2023-07-10T17:08:56.685996+0000 mon.c (mon.1) 75 : audit [INF] from='client.? 172.21.15.116:0/2387753186' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1382514600"}]: access denied 2023-07-10T17:08:59.360 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:58 smithi192 bash[23730]: cluster 2023-07-10T17:08:57.861279+0000 mgr.y (mgr.15252) 28 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:59.361 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:08:58 smithi192 systemd[1]: Started Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:08:59.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:58 smithi116 bash[23740]: cluster 2023-07-10T17:08:57.861279+0000 mgr.y (mgr.15252) 28 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:08:59.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:58 smithi116 bash[28351]: cluster 2023-07-10T17:08:57.861279+0000 mgr.y (mgr.15252) 28 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:00.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:59 smithi192 bash[23730]: audit 2023-07-10T17:08:58.953535+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:00.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:59 smithi192 bash[23730]: audit 2023-07-10T17:08:58.963334+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:00.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:59 smithi192 bash[23730]: audit 2023-07-10T17:08:59.020659+0000 mon.c (mon.1) 76 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:09:00.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:59 smithi192 bash[23730]: audit 2023-07-10T17:08:59.022202+0000 mgr.y (mgr.15252) 29 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:09:00.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:59 smithi192 bash[23730]: audit 2023-07-10T17:08:59.032641+0000 mon.c (mon.1) 77 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:09:00.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:59 smithi192 bash[23730]: audit 2023-07-10T17:08:59.033745+0000 mgr.y (mgr.15252) 30 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:09:00.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:59 smithi192 bash[23730]: audit 2023-07-10T17:08:59.044822+0000 mon.c (mon.1) 78 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:09:00.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:59 smithi192 bash[23730]: audit 2023-07-10T17:08:59.045953+0000 mgr.y (mgr.15252) 31 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:09:00.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:08:59 smithi192 bash[23730]: audit 2023-07-10T17:08:59.054656+0000 mon.c (mon.1) 79 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:09:00.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[23740]: audit 2023-07-10T17:08:58.953535+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:00.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[23740]: audit 2023-07-10T17:08:58.963334+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:00.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[23740]: audit 2023-07-10T17:08:59.020659+0000 mon.c (mon.1) 76 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:09:00.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[23740]: audit 2023-07-10T17:08:59.022202+0000 mgr.y (mgr.15252) 29 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:09:00.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[23740]: audit 2023-07-10T17:08:59.032641+0000 mon.c (mon.1) 77 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:09:00.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[23740]: audit 2023-07-10T17:08:59.033745+0000 mgr.y (mgr.15252) 30 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:09:00.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[23740]: audit 2023-07-10T17:08:59.044822+0000 mon.c (mon.1) 78 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:09:00.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[23740]: audit 2023-07-10T17:08:59.045953+0000 mgr.y (mgr.15252) 31 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:09:00.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[23740]: audit 2023-07-10T17:08:59.054656+0000 mon.c (mon.1) 79 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:09:00.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[28351]: audit 2023-07-10T17:08:58.953535+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:00.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[28351]: audit 2023-07-10T17:08:58.963334+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:00.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[28351]: audit 2023-07-10T17:08:59.020659+0000 mon.c (mon.1) 76 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:09:00.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[28351]: audit 2023-07-10T17:08:59.022202+0000 mgr.y (mgr.15252) 29 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:09:00.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[28351]: audit 2023-07-10T17:08:59.032641+0000 mon.c (mon.1) 77 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:09:00.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[28351]: audit 2023-07-10T17:08:59.033745+0000 mgr.y (mgr.15252) 30 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:09:00.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[28351]: audit 2023-07-10T17:08:59.044822+0000 mon.c (mon.1) 78 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:09:00.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[28351]: audit 2023-07-10T17:08:59.045953+0000 mgr.y (mgr.15252) 31 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:09:00.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:08:59 smithi116 bash[28351]: audit 2023-07-10T17:08:59.054656+0000 mon.c (mon.1) 79 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:09:00.778 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.609Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-10T17:09:00.779 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.609Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-10T17:09:00.779 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.609Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-10T17:09:00.779 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.609Z caller=main.go:339 host_details="(Linux 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 smithi192 (none))" 2023-07-10T17:09:00.779 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.609Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-10T17:09:00.779 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.610Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-10T17:09:00.780 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.611Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-10T17:09:00.780 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.611Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-10T17:09:00.780 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.615Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-10T17:09:00.780 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.615Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=7 2023-07-10T17:09:00.781 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.628Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=7 2023-07-10T17:09:00.781 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.629Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=7 2023-07-10T17:09:00.781 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.629Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=7 2023-07-10T17:09:00.781 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.629Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=7 2023-07-10T17:09:00.781 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.629Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=7 2023-07-10T17:09:00.781 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.630Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=7 2023-07-10T17:09:00.781 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.630Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=7 2023-07-10T17:09:00.782 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.630Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=15.124109ms 2023-07-10T17:09:00.782 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.633Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-10T17:09:00.782 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.633Z caller=main.go:695 msg="TSDB started" 2023-07-10T17:09:00.782 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.633Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-10T17:09:00.783 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.633Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-10T17:09:00.783 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.633Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-10T17:09:00.783 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.633Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-10T17:09:00.783 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.633Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-10T17:09:00.783 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.633Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-10T17:09:00.783 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.633Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-10T17:09:00.784 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.633Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-10T17:09:00.784 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.633Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-10T17:09:00.784 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.634Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-10T17:09:00.784 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=info ts=2023-07-10T17:09:00.634Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-10T17:09:00.784 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[47008]: level=error ts=2023-07-10T17:09:00.634Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-10T17:09:01.084 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:00 smithi192 bash[23730]: cluster 2023-07-10T17:08:59.862882+0000 mgr.y (mgr.15252) 32 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-07-10T17:09:01.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:00 smithi116 bash[28351]: cluster 2023-07-10T17:08:59.862882+0000 mgr.y (mgr.15252) 32 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-07-10T17:09:01.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:00 smithi116 bash[23740]: cluster 2023-07-10T17:08:59.862882+0000 mgr.y (mgr.15252) 32 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-07-10T17:09:01.978 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:01 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-10T17:09:01.978 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:01 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-10T17:09:02.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:01 smithi192 bash[23730]: audit 2023-07-10T17:09:01.832476+0000 mon.c (mon.1) 80 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:09:02.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:01 smithi192 bash[23730]: audit 2023-07-10T17:09:01.833365+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:09:02.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:01 smithi192 bash[23730]: audit 2023-07-10T17:09:01.835908+0000 mon.c (mon.1) 81 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T17:09:02.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:01 smithi192 bash[23730]: audit 2023-07-10T17:09:01.838222+0000 mon.c (mon.1) 82 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:09:02.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:01 smithi116 bash[23740]: audit 2023-07-10T17:09:01.832476+0000 mon.c (mon.1) 80 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:09:02.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:01 smithi116 bash[23740]: audit 2023-07-10T17:09:01.833365+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:09:02.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:01 smithi116 bash[23740]: audit 2023-07-10T17:09:01.835908+0000 mon.c (mon.1) 81 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T17:09:02.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:01 smithi116 bash[23740]: audit 2023-07-10T17:09:01.838222+0000 mon.c (mon.1) 82 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:09:02.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:01 smithi116 bash[28351]: audit 2023-07-10T17:09:01.832476+0000 mon.c (mon.1) 80 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:09:02.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:01 smithi116 bash[28351]: audit 2023-07-10T17:09:01.833365+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:09:02.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:01 smithi116 bash[28351]: audit 2023-07-10T17:09:01.835908+0000 mon.c (mon.1) 81 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T17:09:02.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:01 smithi116 bash[28351]: audit 2023-07-10T17:09:01.838222+0000 mon.c (mon.1) 82 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:09:03.346 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:02 smithi192 bash[23730]: cephadm 2023-07-10T17:09:01.831647+0000 mgr.y (mgr.15252) 33 : cephadm [INF] Upgrade: Updating mgr.x 2023-07-10T17:09:03.347 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:02 smithi192 bash[23730]: cephadm 2023-07-10T17:09:01.841176+0000 mgr.y (mgr.15252) 34 : cephadm [INF] Deploying daemon mgr.x on smithi192 2023-07-10T17:09:03.347 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:02 smithi192 bash[23730]: cluster 2023-07-10T17:09:01.863732+0000 mgr.y (mgr.15252) 35 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-07-10T17:09:03.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:02 smithi116 bash[23740]: cephadm 2023-07-10T17:09:01.831647+0000 mgr.y (mgr.15252) 33 : cephadm [INF] Upgrade: Updating mgr.x 2023-07-10T17:09:03.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:02 smithi116 bash[23740]: cephadm 2023-07-10T17:09:01.841176+0000 mgr.y (mgr.15252) 34 : cephadm [INF] Deploying daemon mgr.x on smithi192 2023-07-10T17:09:03.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:02 smithi116 bash[23740]: cluster 2023-07-10T17:09:01.863732+0000 mgr.y (mgr.15252) 35 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-07-10T17:09:03.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:02 smithi116 bash[28351]: cephadm 2023-07-10T17:09:01.831647+0000 mgr.y (mgr.15252) 33 : cephadm [INF] Upgrade: Updating mgr.x 2023-07-10T17:09:03.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:02 smithi116 bash[28351]: cephadm 2023-07-10T17:09:01.841176+0000 mgr.y (mgr.15252) 34 : cephadm [INF] Deploying daemon mgr.x on smithi192 2023-07-10T17:09:03.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:02 smithi116 bash[28351]: cluster 2023-07-10T17:09:01.863732+0000 mgr.y (mgr.15252) 35 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-07-10T17:09:04.136 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:04 smithi192 systemd[1]: Stopping Ceph mgr.x for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:09:04.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:04 smithi192 bash[23730]: cluster 2023-07-10T17:09:03.864660+0000 mgr.y (mgr.15252) 36 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:04.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:04 smithi116 bash[23740]: cluster 2023-07-10T17:09:03.864660+0000 mgr.y (mgr.15252) 36 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:04.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:04 smithi116 bash[28351]: cluster 2023-07-10T17:09:03.864660+0000 mgr.y (mgr.15252) 36 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:05.136 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:04 smithi192 bash[47336]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-mgr.x 2023-07-10T17:09:05.137 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:04 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.x.service: Main process exited, code=exited, status=143/n/a 2023-07-10T17:09:05.137 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:04 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.x.service: Failed with result 'exit-code'. 2023-07-10T17:09:05.137 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:04 smithi192 systemd[1]: Stopped Ceph mgr.x for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:09:05.636 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:05 smithi192 systemd[1]: Started Ceph mgr.x for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:09:06.586 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:06 smithi192 bash[23730]: audit 2023-07-10T17:09:05.270665+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:06.586 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:06 smithi192 bash[23730]: audit 2023-07-10T17:09:05.282000+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:06.587 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:06 smithi192 bash[47487]: debug 2023-07-10T17:09:06.415+0000 7f418260d700 1 -- 172.21.15.192:0/1112610383 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5600affbf1e0 con 0x5600b0d3e800 2023-07-10T17:09:06.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:06 smithi116 bash[23740]: audit 2023-07-10T17:09:05.270665+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:06.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:06 smithi116 bash[23740]: audit 2023-07-10T17:09:05.282000+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:06.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:06 smithi116 bash[28351]: audit 2023-07-10T17:09:05.270665+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:06.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:06 smithi116 bash[28351]: audit 2023-07-10T17:09:05.282000+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:06.886 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:06 smithi192 bash[47487]: debug 2023-07-10T17:09:06.579+0000 7f418c75b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-10T17:09:07.559 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:07 smithi192 bash[23730]: cluster 2023-07-10T17:09:05.866443+0000 mgr.y (mgr.15252) 37 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:07.560 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:07 smithi192 bash[47487]: debug 2023-07-10T17:09:07.451+0000 7f418c75b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-10T17:09:07.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:07 smithi116 bash[23740]: cluster 2023-07-10T17:09:05.866443+0000 mgr.y (mgr.15252) 37 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:07.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:07 smithi116 bash[28351]: cluster 2023-07-10T17:09:05.866443+0000 mgr.y (mgr.15252) 37 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:07.886 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:07 smithi192 bash[47487]: debug 2023-07-10T17:09:07.551+0000 7f418c75b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-10T17:09:07.887 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:07 smithi192 bash[47487]: debug 2023-07-10T17:09:07.655+0000 7f418c75b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-10T17:09:08.264 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:07 smithi192 bash[47487]: debug 2023-07-10T17:09:07.915+0000 7f418c75b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-10T17:09:08.629 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:08 smithi192 bash[47487]: debug 2023-07-10T17:09:08.255+0000 7f418c75b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-10T17:09:08.629 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:08 smithi192 bash[47487]: debug 2023-07-10T17:09:08.415+0000 7f418c75b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-10T17:09:08.886 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:08 smithi192 bash[47487]: debug 2023-07-10T17:09:08.623+0000 7f418c75b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-10T17:09:09.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:09 smithi192 bash[23730]: cluster 2023-07-10T17:09:07.867265+0000 mgr.y (mgr.15252) 38 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:09.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:09 smithi116 bash[23740]: cluster 2023-07-10T17:09:07.867265+0000 mgr.y (mgr.15252) 38 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:09.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:09 smithi116 bash[28351]: cluster 2023-07-10T17:09:07.867265+0000 mgr.y (mgr.15252) 38 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:10.136 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:09 smithi192 bash[47487]: debug 2023-07-10T17:09:09.675+0000 7f418c75b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-10T17:09:10.137 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:09 smithi192 bash[47487]: debug 2023-07-10T17:09:09.791+0000 7f418c75b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-10T17:09:10.137 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:09 smithi192 bash[47487]: debug 2023-07-10T17:09:09.907+0000 7f418c75b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-10T17:09:10.642 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:10 smithi192 bash[47487]: debug 2023-07-10T17:09:10.383+0000 7f418c75b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-10T17:09:11.049 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:10 smithi192 bash[47487]: debug 2023-07-10T17:09:10.635+0000 7f418c75b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-10T17:09:11.301 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:11 smithi192 bash[47487]: debug 2023-07-10T17:09:11.043+0000 7f418c75b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-10T17:09:11.303 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:11 smithi192 bash[23730]: cluster 2023-07-10T17:09:09.868849+0000 mgr.y (mgr.15252) 39 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:11.304 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:11 smithi192 bash[23730]: audit 2023-07-10T17:09:10.996175+0000 mon.c (mon.1) 83 : audit [DBG] from='client.? 172.21.15.116:0/2990005888' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:09:11.591 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:11 smithi116 bash[28351]: cluster 2023-07-10T17:09:09.868849+0000 mgr.y (mgr.15252) 39 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:11.591 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:11 smithi116 bash[28351]: audit 2023-07-10T17:09:10.996175+0000 mon.c (mon.1) 83 : audit [DBG] from='client.? 172.21.15.116:0/2990005888' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:09:11.591 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:11 smithi116 bash[23740]: cluster 2023-07-10T17:09:09.868849+0000 mgr.y (mgr.15252) 39 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:11.592 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:11 smithi116 bash[23740]: audit 2023-07-10T17:09:10.996175+0000 mon.c (mon.1) 83 : audit [DBG] from='client.? 172.21.15.116:0/2990005888' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:09:11.886 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:11 smithi192 bash[47487]: debug 2023-07-10T17:09:11.579+0000 7f418c75b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-10T17:09:12.186 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:11 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 1. 2023-07-10T17:09:12.187 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:11 smithi192 systemd[1]: Stopped Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:09:12.188 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:11 smithi192 systemd[1]: Started Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:09:12.482 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:12 smithi116 bash[23740]: audit 2023-07-10T17:09:11.370944+0000 mon.a (mon.0) 788 : audit [INF] from='client.? 172.21.15.116:0/3604594730' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1382514600"}]: access denied 2023-07-10T17:09:12.483 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:12 smithi116 bash[23740]: audit 2023-07-10T17:09:11.759854+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:12.483 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:12 smithi116 bash[23740]: audit 2023-07-10T17:09:11.772548+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:12.483 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:12 smithi116 bash[28351]: audit 2023-07-10T17:09:11.370944+0000 mon.a (mon.0) 788 : audit [INF] from='client.? 172.21.15.116:0/3604594730' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1382514600"}]: access denied 2023-07-10T17:09:12.483 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:12 smithi116 bash[28351]: audit 2023-07-10T17:09:11.759854+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:12.483 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:12 smithi116 bash[28351]: audit 2023-07-10T17:09:11.772548+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:12.636 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:12 smithi192 bash[47487]: debug 2023-07-10T17:09:12.175+0000 7f418c75b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-10T17:09:12.637 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:12 smithi192 bash[47487]: debug 2023-07-10T17:09:12.291+0000 7f418c75b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-10T17:09:12.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:12 smithi192 bash[23730]: audit 2023-07-10T17:09:11.370944+0000 mon.a (mon.0) 788 : audit [INF] from='client.? 172.21.15.116:0/3604594730' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1382514600"}]: access denied 2023-07-10T17:09:12.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:12 smithi192 bash[23730]: audit 2023-07-10T17:09:11.759854+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:12.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:12 smithi192 bash[23730]: audit 2023-07-10T17:09:11.772548+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:12.973 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:12 smithi192 bash[47487]: debug 2023-07-10T17:09:12.683+0000 7f418c75b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-10T17:09:12.973 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:12 smithi192 bash[47487]: debug 2023-07-10T17:09:12.811+0000 7f418c75b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-10T17:09:13.224 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:12 smithi192 bash[47487]: debug 2023-07-10T17:09:12.967+0000 7f418c75b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-10T17:09:13.224 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[47487]: debug 2023-07-10T17:09:13.091+0000 7f418c75b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-10T17:09:13.224 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.200Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-10T17:09:13.225 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.200Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-10T17:09:13.225 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.200Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-10T17:09:13.225 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.200Z caller=main.go:339 host_details="(Linux 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 smithi192 (none))" 2023-07-10T17:09:13.225 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.200Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-10T17:09:13.225 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.200Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-10T17:09:13.225 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.202Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-10T17:09:13.225 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.202Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-10T17:09:13.226 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.205Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-10T17:09:13.226 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.205Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=8 2023-07-10T17:09:13.226 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.216Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=8 2023-07-10T17:09:13.226 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.216Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=8 2023-07-10T17:09:13.226 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.216Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=8 2023-07-10T17:09:13.226 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.217Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=8 2023-07-10T17:09:13.226 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.217Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=8 2023-07-10T17:09:13.227 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.217Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=8 2023-07-10T17:09:13.227 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.217Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=8 2023-07-10T17:09:13.227 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.218Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=8 2023-07-10T17:09:13.227 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.218Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=12.584597ms 2023-07-10T17:09:13.227 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.220Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-10T17:09:13.227 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.220Z caller=main.go:695 msg="TSDB started" 2023-07-10T17:09:13.227 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.220Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-10T17:09:13.227 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.221Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-10T17:09:13.228 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.221Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-10T17:09:13.228 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.221Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-10T17:09:13.228 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.221Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-10T17:09:13.228 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.221Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-10T17:09:13.228 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.221Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-10T17:09:13.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[23730]: cluster 2023-07-10T17:09:11.869418+0000 mgr.y (mgr.15252) 40 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:13.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[23730]: audit 2023-07-10T17:09:12.314924+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:13.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[23730]: audit 2023-07-10T17:09:12.326954+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:13.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[23730]: audit 2023-07-10T17:09:12.520167+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:13.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[23730]: audit 2023-07-10T17:09:12.533491+0000 mon.a (mon.0) 794 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:13.592 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[47487]: debug 2023-07-10T17:09:13.247+0000 7f418c75b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-10T17:09:13.592 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[47487]: debug 2023-07-10T17:09:13.451+0000 7f418c75b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-10T17:09:13.592 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.221Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-10T17:09:13.592 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.221Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-10T17:09:13.592 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.221Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-10T17:09:13.592 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=info ts=2023-07-10T17:09:13.221Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-10T17:09:13.593 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[48188]: level=error ts=2023-07-10T17:09:13.221Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-10T17:09:13.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:13 smithi116 bash[23740]: cluster 2023-07-10T17:09:11.869418+0000 mgr.y (mgr.15252) 40 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:13.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:13 smithi116 bash[23740]: audit 2023-07-10T17:09:12.314924+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:13.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:13 smithi116 bash[23740]: audit 2023-07-10T17:09:12.326954+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:13.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:13 smithi116 bash[23740]: audit 2023-07-10T17:09:12.520167+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:13.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:13 smithi116 bash[23740]: audit 2023-07-10T17:09:12.533491+0000 mon.a (mon.0) 794 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:13.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:13 smithi116 bash[28351]: cluster 2023-07-10T17:09:11.869418+0000 mgr.y (mgr.15252) 40 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:13.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:13 smithi116 bash[28351]: audit 2023-07-10T17:09:12.314924+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:13.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:13 smithi116 bash[28351]: audit 2023-07-10T17:09:12.326954+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:13.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:13 smithi116 bash[28351]: audit 2023-07-10T17:09:12.520167+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:13.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:13 smithi116 bash[28351]: audit 2023-07-10T17:09:12.533491+0000 mon.a (mon.0) 794 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:13.887 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:13 smithi192 bash[47487]: debug 2023-07-10T17:09:13.583+0000 7f418c75b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-10T17:09:14.386 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-10T17:09:14.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:13 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-10T17:09:14.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:14 smithi192 bash[23730]: cluster 2023-07-10T17:09:13.870129+0000 mgr.y (mgr.15252) 41 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:14.887 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:14 smithi192 bash[47487]: debug 2023-07-10T17:09:14.823+0000 7f418c75b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-10T17:09:14.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:14 smithi116 bash[28351]: cluster 2023-07-10T17:09:13.870129+0000 mgr.y (mgr.15252) 41 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:14.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:14 smithi116 bash[23740]: cluster 2023-07-10T17:09:13.870129+0000 mgr.y (mgr.15252) 41 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:15.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:15 smithi192 bash[23730]: cluster 2023-07-10T17:09:14.849138+0000 mon.a (mon.0) 795 : cluster [DBG] Standby manager daemon x restarted 2023-07-10T17:09:15.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:15 smithi192 bash[23730]: cluster 2023-07-10T17:09:14.849390+0000 mon.a (mon.0) 796 : cluster [DBG] Standby manager daemon x started 2023-07-10T17:09:15.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:15 smithi192 bash[23730]: audit 2023-07-10T17:09:14.851723+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.? 172.21.15.192:0/2509654625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-10T17:09:15.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:15 smithi192 bash[23730]: audit 2023-07-10T17:09:14.852870+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.? 172.21.15.192:0/2509654625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:09:15.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:15 smithi192 bash[23730]: audit 2023-07-10T17:09:14.854966+0000 mon.b (mon.2) 83 : audit [DBG] from='mgr.? 172.21.15.192:0/2509654625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-10T17:09:15.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:15 smithi192 bash[23730]: audit 2023-07-10T17:09:14.855634+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.? 172.21.15.192:0/2509654625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:09:15.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:15 smithi116 bash[28351]: cluster 2023-07-10T17:09:14.849138+0000 mon.a (mon.0) 795 : cluster [DBG] Standby manager daemon x restarted 2023-07-10T17:09:15.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:15 smithi116 bash[28351]: cluster 2023-07-10T17:09:14.849390+0000 mon.a (mon.0) 796 : cluster [DBG] Standby manager daemon x started 2023-07-10T17:09:15.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:15 smithi116 bash[28351]: audit 2023-07-10T17:09:14.851723+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.? 172.21.15.192:0/2509654625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-10T17:09:15.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:15 smithi116 bash[28351]: audit 2023-07-10T17:09:14.852870+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.? 172.21.15.192:0/2509654625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:09:15.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:15 smithi116 bash[28351]: audit 2023-07-10T17:09:14.854966+0000 mon.b (mon.2) 83 : audit [DBG] from='mgr.? 172.21.15.192:0/2509654625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-10T17:09:15.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:15 smithi116 bash[28351]: audit 2023-07-10T17:09:14.855634+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.? 172.21.15.192:0/2509654625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:09:15.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:15 smithi116 bash[23740]: cluster 2023-07-10T17:09:14.849138+0000 mon.a (mon.0) 795 : cluster [DBG] Standby manager daemon x restarted 2023-07-10T17:09:15.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:15 smithi116 bash[23740]: cluster 2023-07-10T17:09:14.849390+0000 mon.a (mon.0) 796 : cluster [DBG] Standby manager daemon x started 2023-07-10T17:09:15.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:15 smithi116 bash[23740]: audit 2023-07-10T17:09:14.851723+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.? 172.21.15.192:0/2509654625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-10T17:09:15.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:15 smithi116 bash[23740]: audit 2023-07-10T17:09:14.852870+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.? 172.21.15.192:0/2509654625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:09:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:15 smithi116 bash[23740]: audit 2023-07-10T17:09:14.854966+0000 mon.b (mon.2) 83 : audit [DBG] from='mgr.? 172.21.15.192:0/2509654625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-10T17:09:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:15 smithi116 bash[23740]: audit 2023-07-10T17:09:14.855634+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.? 172.21.15.192:0/2509654625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:09:16.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:16 smithi192 bash[23730]: cluster 2023-07-10T17:09:15.623212+0000 mon.a (mon.0) 797 : cluster [DBG] mgrmap e26: y(active, since 42s), standbys: x 2023-07-10T17:09:16.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:16 smithi192 bash[23730]: cluster 2023-07-10T17:09:15.871944+0000 mgr.y (mgr.15252) 42 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:16.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:16 smithi116 bash[28351]: cluster 2023-07-10T17:09:15.623212+0000 mon.a (mon.0) 797 : cluster [DBG] mgrmap e26: y(active, since 42s), standbys: x 2023-07-10T17:09:16.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:16 smithi116 bash[28351]: cluster 2023-07-10T17:09:15.871944+0000 mgr.y (mgr.15252) 42 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:16.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:16 smithi116 bash[23740]: cluster 2023-07-10T17:09:15.623212+0000 mon.a (mon.0) 797 : cluster [DBG] mgrmap e26: y(active, since 42s), standbys: x 2023-07-10T17:09:16.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:16 smithi116 bash[23740]: cluster 2023-07-10T17:09:15.871944+0000 mgr.y (mgr.15252) 42 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:19.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:18 smithi192 bash[23730]: cluster 2023-07-10T17:09:17.872799+0000 mgr.y (mgr.15252) 43 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:19.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:18 smithi116 bash[23740]: cluster 2023-07-10T17:09:17.872799+0000 mgr.y (mgr.15252) 43 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:19.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:18 smithi116 bash[28351]: cluster 2023-07-10T17:09:17.872799+0000 mgr.y (mgr.15252) 43 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:21.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:20 smithi192 bash[23730]: cluster 2023-07-10T17:09:19.874436+0000 mgr.y (mgr.15252) 44 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:21.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:20 smithi116 bash[23740]: cluster 2023-07-10T17:09:19.874436+0000 mgr.y (mgr.15252) 44 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:21.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:20 smithi116 bash[28351]: cluster 2023-07-10T17:09:19.874436+0000 mgr.y (mgr.15252) 44 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:23.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: cluster 2023-07-10T17:09:21.875279+0000 mgr.y (mgr.15252) 45 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:23.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: audit 2023-07-10T17:09:22.590895+0000 mon.a (mon.0) 798 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:23.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: audit 2023-07-10T17:09:22.604774+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:23.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: audit 2023-07-10T17:09:22.607896+0000 mon.c (mon.1) 84 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:09:23.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: audit 2023-07-10T17:09:22.609634+0000 mon.c (mon.1) 85 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:09:23.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: audit 2023-07-10T17:09:22.620959+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:23.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: audit 2023-07-10T17:09:22.708200+0000 mon.c (mon.1) 86 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:09:23.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: audit 2023-07-10T17:09:22.713804+0000 mon.c (mon.1) 87 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:09:23.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: audit 2023-07-10T17:09:22.724719+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:23.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: audit 2023-07-10T17:09:22.727895+0000 mon.c (mon.1) 88 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2023-07-10T17:09:23.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: audit 2023-07-10T17:09:22.728380+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2023-07-10T17:09:23.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: audit 2023-07-10T17:09:22.736702+0000 mon.a (mon.0) 803 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2023-07-10T17:09:23.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: audit 2023-07-10T17:09:22.738367+0000 mon.c (mon.1) 89 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2023-07-10T17:09:23.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: audit 2023-07-10T17:09:22.738908+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2023-07-10T17:09:23.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: audit 2023-07-10T17:09:22.745734+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2023-07-10T17:09:23.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: audit 2023-07-10T17:09:22.748732+0000 mon.c (mon.1) 90 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-10T17:09:23.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:22 smithi192 bash[23730]: audit 2023-07-10T17:09:22.750432+0000 mon.c (mon.1) 91 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2023-07-10T17:09:23.420 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: cluster 2023-07-10T17:09:21.875279+0000 mgr.y (mgr.15252) 45 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:23.420 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: audit 2023-07-10T17:09:22.590895+0000 mon.a (mon.0) 798 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:23.421 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: audit 2023-07-10T17:09:22.604774+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:23.421 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: audit 2023-07-10T17:09:22.607896+0000 mon.c (mon.1) 84 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:09:23.421 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: audit 2023-07-10T17:09:22.609634+0000 mon.c (mon.1) 85 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:09:23.421 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: audit 2023-07-10T17:09:22.620959+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:23.421 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: audit 2023-07-10T17:09:22.708200+0000 mon.c (mon.1) 86 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:09:23.421 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: audit 2023-07-10T17:09:22.713804+0000 mon.c (mon.1) 87 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:09:23.421 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: audit 2023-07-10T17:09:22.724719+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:23.422 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: audit 2023-07-10T17:09:22.727895+0000 mon.c (mon.1) 88 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2023-07-10T17:09:23.422 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: audit 2023-07-10T17:09:22.728380+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2023-07-10T17:09:23.422 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: audit 2023-07-10T17:09:22.736702+0000 mon.a (mon.0) 803 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2023-07-10T17:09:23.422 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: audit 2023-07-10T17:09:22.738367+0000 mon.c (mon.1) 89 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2023-07-10T17:09:23.422 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: audit 2023-07-10T17:09:22.738908+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2023-07-10T17:09:23.422 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: audit 2023-07-10T17:09:22.745734+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2023-07-10T17:09:23.422 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: audit 2023-07-10T17:09:22.748732+0000 mon.c (mon.1) 90 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-10T17:09:23.422 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[28351]: audit 2023-07-10T17:09:22.750432+0000 mon.c (mon.1) 91 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2023-07-10T17:09:23.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: cluster 2023-07-10T17:09:21.875279+0000 mgr.y (mgr.15252) 45 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:23.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: audit 2023-07-10T17:09:22.590895+0000 mon.a (mon.0) 798 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:23.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: audit 2023-07-10T17:09:22.604774+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:23.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: audit 2023-07-10T17:09:22.607896+0000 mon.c (mon.1) 84 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:09:23.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: audit 2023-07-10T17:09:22.609634+0000 mon.c (mon.1) 85 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:09:23.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: audit 2023-07-10T17:09:22.620959+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:23.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: audit 2023-07-10T17:09:22.708200+0000 mon.c (mon.1) 86 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:09:23.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: audit 2023-07-10T17:09:22.713804+0000 mon.c (mon.1) 87 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:09:23.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: audit 2023-07-10T17:09:22.724719+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:23.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: audit 2023-07-10T17:09:22.727895+0000 mon.c (mon.1) 88 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2023-07-10T17:09:23.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: audit 2023-07-10T17:09:22.728380+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2023-07-10T17:09:23.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: audit 2023-07-10T17:09:22.736702+0000 mon.a (mon.0) 803 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2023-07-10T17:09:23.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: audit 2023-07-10T17:09:22.738367+0000 mon.c (mon.1) 89 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2023-07-10T17:09:23.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: audit 2023-07-10T17:09:22.738908+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2023-07-10T17:09:23.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: audit 2023-07-10T17:09:22.745734+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.15252 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2023-07-10T17:09:23.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: audit 2023-07-10T17:09:22.748732+0000 mon.c (mon.1) 90 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-10T17:09:23.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:22 smithi116 bash[23740]: audit 2023-07-10T17:09:22.750432+0000 mon.c (mon.1) 91 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2023-07-10T17:09:24.204 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:24 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 2. 2023-07-10T17:09:24.204 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:24 smithi192 systemd[1]: Stopped Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:09:24.205 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:24 smithi192 systemd[1]: Started Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:09:24.205 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:23 smithi192 bash[23730]: cephadm 2023-07-10T17:09:22.611122+0000 mgr.y (mgr.15252) 46 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:09:24.205 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:23 smithi192 bash[23730]: cluster 2023-07-10T17:09:22.612339+0000 mgr.y (mgr.15252) 47 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 95 B/s rd, 0 op/s 2023-07-10T17:09:24.205 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:23 smithi192 bash[23730]: cluster 2023-07-10T17:09:22.612887+0000 mgr.y (mgr.15252) 48 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 117 B/s rd, 0 op/s 2023-07-10T17:09:24.205 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:23 smithi192 bash[23730]: cephadm 2023-07-10T17:09:22.715387+0000 mgr.y (mgr.15252) 49 : cephadm [INF] Upgrade: Setting container_image for all mgr 2023-07-10T17:09:24.205 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:23 smithi192 bash[23730]: cephadm 2023-07-10T17:09:22.751620+0000 mgr.y (mgr.15252) 50 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2023-07-10T17:09:24.206 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:23 smithi192 bash[23730]: cluster 2023-07-10T17:09:23.601673+0000 mon.a (mon.0) 806 : cluster [WRN] Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:09:24.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:23 smithi116 bash[23740]: cephadm 2023-07-10T17:09:22.611122+0000 mgr.y (mgr.15252) 46 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:09:24.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:23 smithi116 bash[23740]: cluster 2023-07-10T17:09:22.612339+0000 mgr.y (mgr.15252) 47 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 95 B/s rd, 0 op/s 2023-07-10T17:09:24.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:23 smithi116 bash[23740]: cluster 2023-07-10T17:09:22.612887+0000 mgr.y (mgr.15252) 48 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 117 B/s rd, 0 op/s 2023-07-10T17:09:24.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:23 smithi116 bash[23740]: cephadm 2023-07-10T17:09:22.715387+0000 mgr.y (mgr.15252) 49 : cephadm [INF] Upgrade: Setting container_image for all mgr 2023-07-10T17:09:24.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:23 smithi116 bash[23740]: cephadm 2023-07-10T17:09:22.751620+0000 mgr.y (mgr.15252) 50 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2023-07-10T17:09:24.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:23 smithi116 bash[23740]: cluster 2023-07-10T17:09:23.601673+0000 mon.a (mon.0) 806 : cluster [WRN] Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:09:24.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:23 smithi116 bash[28351]: cephadm 2023-07-10T17:09:22.611122+0000 mgr.y (mgr.15252) 46 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:09:24.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:23 smithi116 bash[28351]: cluster 2023-07-10T17:09:22.612339+0000 mgr.y (mgr.15252) 47 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 95 B/s rd, 0 op/s 2023-07-10T17:09:24.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:23 smithi116 bash[28351]: cluster 2023-07-10T17:09:22.612887+0000 mgr.y (mgr.15252) 48 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 117 B/s rd, 0 op/s 2023-07-10T17:09:24.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:23 smithi116 bash[28351]: cephadm 2023-07-10T17:09:22.715387+0000 mgr.y (mgr.15252) 49 : cephadm [INF] Upgrade: Setting container_image for all mgr 2023-07-10T17:09:24.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:23 smithi116 bash[28351]: cephadm 2023-07-10T17:09:22.751620+0000 mgr.y (mgr.15252) 50 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2023-07-10T17:09:24.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:23 smithi116 bash[28351]: cluster 2023-07-10T17:09:23.601673+0000 mon.a (mon.0) 806 : cluster [WRN] Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:09:24.589 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:09:25.342 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:09:25.342 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (2m) 13s ago 8m 10.3M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:09:25.342 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (76s) 13s ago 8m 25.3M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:09:25.343 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 running (16s) 13s ago 6m - - 24ecd6d5f14c f12e97cdcef8 2023-07-10T17:09:25.343 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (20s) 13s ago 13m 80.4M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:09:25.343 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (70s) 13s ago 14m 464M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:09:25.343 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (14m) 13s ago 15m 63.6M 2048M 16.2.0 24ecd6d5f14c 33f26d450a67 2023-07-10T17:09:25.343 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (13m) 13s ago 13m 57.5M 2048M 16.2.0 24ecd6d5f14c 9ad4329e5488 2023-07-10T17:09:25.343 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (13m) 13s ago 13m 54.0M 2048M 16.2.0 24ecd6d5f14c 31e3ac12741b 2023-07-10T17:09:25.343 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (2m) 13s ago 9m 9180k - 0.18.1 e5a616e4b9cf 3bd058d91353 2023-07-10T17:09:25.343 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (2m) 13s ago 9m 3439k - 0.18.1 e5a616e4b9cf 63551615c679 2023-07-10T17:09:25.344 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (12m) 13s ago 12m 54.6M 4096M 16.2.0 24ecd6d5f14c 4fa26367c830 2023-07-10T17:09:25.344 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (12m) 13s ago 12m 53.2M 4096M 16.2.0 24ecd6d5f14c c157509b3fd4 2023-07-10T17:09:25.344 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (11m) 13s ago 11m 49.4M 4096M 16.2.0 24ecd6d5f14c f1fd77b3c454 2023-07-10T17:09:25.344 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (11m) 13s ago 11m 54.1M 4096M 16.2.0 24ecd6d5f14c 60949d479c11 2023-07-10T17:09:25.344 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (11m) 13s ago 11m 53.7M 4096M 16.2.0 24ecd6d5f14c 75fc0b36aab8 2023-07-10T17:09:25.344 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (10m) 13s ago 10m 52.1M 4096M 16.2.0 24ecd6d5f14c d673065333dd 2023-07-10T17:09:25.344 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (10m) 13s ago 10m 52.6M 4096M 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:09:25.345 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (9m) 13s ago 9m 55.7M 4096M 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:09:25.345 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 error 13s ago 9m - - 2023-07-10T17:09:25.345 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (6m) 13s ago 6m 53.9M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:09:25.345 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (6m) 13s ago 6m 53.9M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:09:25.345 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (6m) 13s ago 6m 54.1M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:09:25.345 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (6m) 13s ago 6m 55.7M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:09:25.511 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.308Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-10T17:09:25.512 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.308Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-10T17:09:25.512 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.308Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-10T17:09:25.512 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.308Z caller=main.go:339 host_details="(Linux 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 smithi192 (none))" 2023-07-10T17:09:25.513 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.308Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-10T17:09:25.513 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.308Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-10T17:09:25.513 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.312Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-10T17:09:25.514 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.312Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-10T17:09:25.514 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.320Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-10T17:09:25.514 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.320Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=9 2023-07-10T17:09:25.515 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.335Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=9 2023-07-10T17:09:25.515 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.336Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=9 2023-07-10T17:09:25.515 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.336Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=9 2023-07-10T17:09:25.515 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.336Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=9 2023-07-10T17:09:25.515 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.336Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=9 2023-07-10T17:09:25.516 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.339Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=9 2023-07-10T17:09:25.516 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.340Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=9 2023-07-10T17:09:25.516 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.340Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=9 2023-07-10T17:09:25.516 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.340Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=9 2023-07-10T17:09:25.516 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.340Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=20.399378ms 2023-07-10T17:09:25.516 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.343Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-10T17:09:25.516 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.343Z caller=main.go:695 msg="TSDB started" 2023-07-10T17:09:25.517 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.343Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-10T17:09:25.517 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.343Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-10T17:09:25.517 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.343Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-10T17:09:25.517 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.343Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-10T17:09:25.517 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.343Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-10T17:09:25.517 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.343Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-10T17:09:25.517 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.343Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-10T17:09:25.518 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.343Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-10T17:09:25.518 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.343Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-10T17:09:25.518 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.343Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-10T17:09:25.518 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=info ts=2023-07-10T17:09:25.343Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-10T17:09:25.518 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[48595]: level=error ts=2023-07-10T17:09:25.344Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-10T17:09:25.788 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:09:25.788 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:09:25.789 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2023-07-10T17:09:25.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:09:25.789 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:09:25.789 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:09:25.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:09:25.790 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:09:25.790 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-10T17:09:25.790 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:09:25.790 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": {}, 2023-07-10T17:09:25.790 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:09:25.790 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:09:25.790 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:09:25.791 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:09:25.791 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 15, 2023-07-10T17:09:25.791 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:09:25.791 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:09:25.791 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:09:26.149 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:09:26.150 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:09:26.150 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:09:26.150 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:09:26.150 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:09:26.150 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2023-07-10T17:09:26.150 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:09:26.150 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "2/23 daemons upgraded", 2023-07-10T17:09:26.151 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading mon daemons", 2023-07-10T17:09:26.151 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:09:26.151 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:09:26.386 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:26 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-10T17:09:26.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:26 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-10T17:09:26.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[23730]: audit 2023-07-10T17:09:24.568248+0000 mgr.y (mgr.15252) 51 : audit [DBG] from='client.15411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:26.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[23730]: cluster 2023-07-10T17:09:24.613481+0000 mgr.y (mgr.15252) 52 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:26.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[23730]: audit 2023-07-10T17:09:24.940130+0000 mgr.y (mgr.15252) 53 : audit [DBG] from='client.15417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:26.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[23730]: audit 2023-07-10T17:09:25.310722+0000 mgr.y (mgr.15252) 54 : audit [DBG] from='client.25336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:26.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:25 smithi192 bash[23730]: audit 2023-07-10T17:09:25.784979+0000 mon.a (mon.0) 807 : audit [DBG] from='client.? 172.21.15.116:0/1674476497' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:09:26.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:25 smithi116 bash[28351]: audit 2023-07-10T17:09:24.568248+0000 mgr.y (mgr.15252) 51 : audit [DBG] from='client.15411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:26.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:25 smithi116 bash[28351]: cluster 2023-07-10T17:09:24.613481+0000 mgr.y (mgr.15252) 52 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:26.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:25 smithi116 bash[28351]: audit 2023-07-10T17:09:24.940130+0000 mgr.y (mgr.15252) 53 : audit [DBG] from='client.15417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:26.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:25 smithi116 bash[28351]: audit 2023-07-10T17:09:25.310722+0000 mgr.y (mgr.15252) 54 : audit [DBG] from='client.25336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:26.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:25 smithi116 bash[28351]: audit 2023-07-10T17:09:25.784979+0000 mon.a (mon.0) 807 : audit [DBG] from='client.? 172.21.15.116:0/1674476497' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:09:26.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:25 smithi116 bash[23740]: audit 2023-07-10T17:09:24.568248+0000 mgr.y (mgr.15252) 51 : audit [DBG] from='client.15411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:26.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:25 smithi116 bash[23740]: cluster 2023-07-10T17:09:24.613481+0000 mgr.y (mgr.15252) 52 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:26.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:25 smithi116 bash[23740]: audit 2023-07-10T17:09:24.940130+0000 mgr.y (mgr.15252) 53 : audit [DBG] from='client.15417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:26.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:25 smithi116 bash[23740]: audit 2023-07-10T17:09:25.310722+0000 mgr.y (mgr.15252) 54 : audit [DBG] from='client.25336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:26.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:25 smithi116 bash[23740]: audit 2023-07-10T17:09:25.784979+0000 mon.a (mon.0) 807 : audit [DBG] from='client.? 172.21.15.116:0/1674476497' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:09:26.604 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2023-07-10T17:09:26.605 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2023-07-10T17:09:26.605 INFO:teuthology.orchestra.run.smithi116.stdout: daemon prometheus.a on smithi192 is in error state 2023-07-10T17:09:27.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:26 smithi192 bash[23730]: audit 2023-07-10T17:09:26.110896+0000 mon.c (mon.1) 92 : audit [DBG] from='client.? 172.21.15.116:0/662119364' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:09:27.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:26 smithi192 bash[23730]: audit 2023-07-10T17:09:26.145878+0000 mgr.y (mgr.15252) 55 : audit [DBG] from='client.25366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:27.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:26 smithi192 bash[23730]: cephadm 2023-07-10T17:09:26.328487+0000 mgr.y (mgr.15252) 56 : cephadm [INF] Upgrade: Updating mon.a 2023-07-10T17:09:27.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:26 smithi192 bash[23730]: audit 2023-07-10T17:09:26.337676+0000 mon.a (mon.0) 808 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:27.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:26 smithi192 bash[23730]: audit 2023-07-10T17:09:26.339990+0000 mon.c (mon.1) 93 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:09:27.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:26 smithi192 bash[23730]: audit 2023-07-10T17:09:26.341142+0000 mon.c (mon.1) 94 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:09:27.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:26 smithi192 bash[23730]: audit 2023-07-10T17:09:26.342224+0000 mon.c (mon.1) 95 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:09:27.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:26 smithi192 bash[23730]: cephadm 2023-07-10T17:09:26.343243+0000 mgr.y (mgr.15252) 57 : cephadm [INF] Deploying daemon mon.a on smithi116 2023-07-10T17:09:27.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:26 smithi192 bash[23730]: audit 2023-07-10T17:09:26.490443+0000 mon.a (mon.0) 809 : audit [INF] from='client.? 172.21.15.116:0/1077768123' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1382514600"}]: access denied 2023-07-10T17:09:27.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:26 smithi192 bash[23730]: audit 2023-07-10T17:09:26.600914+0000 mon.a (mon.0) 810 : audit [DBG] from='client.? 172.21.15.116:0/1800160153' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:09:27.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[23740]: audit 2023-07-10T17:09:26.110896+0000 mon.c (mon.1) 92 : audit [DBG] from='client.? 172.21.15.116:0/662119364' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:09:27.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[23740]: audit 2023-07-10T17:09:26.145878+0000 mgr.y (mgr.15252) 55 : audit [DBG] from='client.25366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:27.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[23740]: cephadm 2023-07-10T17:09:26.328487+0000 mgr.y (mgr.15252) 56 : cephadm [INF] Upgrade: Updating mon.a 2023-07-10T17:09:27.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[23740]: audit 2023-07-10T17:09:26.337676+0000 mon.a (mon.0) 808 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:27.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[23740]: audit 2023-07-10T17:09:26.339990+0000 mon.c (mon.1) 93 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:09:27.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[23740]: audit 2023-07-10T17:09:26.341142+0000 mon.c (mon.1) 94 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:09:27.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[23740]: audit 2023-07-10T17:09:26.342224+0000 mon.c (mon.1) 95 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:09:27.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[23740]: cephadm 2023-07-10T17:09:26.343243+0000 mgr.y (mgr.15252) 57 : cephadm [INF] Deploying daemon mon.a on smithi116 2023-07-10T17:09:27.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[23740]: audit 2023-07-10T17:09:26.490443+0000 mon.a (mon.0) 809 : audit [INF] from='client.? 172.21.15.116:0/1077768123' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1382514600"}]: access denied 2023-07-10T17:09:27.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[23740]: audit 2023-07-10T17:09:26.600914+0000 mon.a (mon.0) 810 : audit [DBG] from='client.? 172.21.15.116:0/1800160153' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:09:27.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[28351]: audit 2023-07-10T17:09:26.110896+0000 mon.c (mon.1) 92 : audit [DBG] from='client.? 172.21.15.116:0/662119364' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:09:27.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[28351]: audit 2023-07-10T17:09:26.145878+0000 mgr.y (mgr.15252) 55 : audit [DBG] from='client.25366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:27.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[28351]: cephadm 2023-07-10T17:09:26.328487+0000 mgr.y (mgr.15252) 56 : cephadm [INF] Upgrade: Updating mon.a 2023-07-10T17:09:27.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[28351]: audit 2023-07-10T17:09:26.337676+0000 mon.a (mon.0) 808 : audit [INF] from='mgr.15252 ' entity='mgr.y' 2023-07-10T17:09:27.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[28351]: audit 2023-07-10T17:09:26.339990+0000 mon.c (mon.1) 93 : audit [INF] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:09:27.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[28351]: audit 2023-07-10T17:09:26.341142+0000 mon.c (mon.1) 94 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:09:27.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[28351]: audit 2023-07-10T17:09:26.342224+0000 mon.c (mon.1) 95 : audit [DBG] from='mgr.15252 172.21.15.116:0/3666593709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:09:27.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[28351]: cephadm 2023-07-10T17:09:26.343243+0000 mgr.y (mgr.15252) 57 : cephadm [INF] Deploying daemon mon.a on smithi116 2023-07-10T17:09:27.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[28351]: audit 2023-07-10T17:09:26.490443+0000 mon.a (mon.0) 809 : audit [INF] from='client.? 172.21.15.116:0/1077768123' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1382514600"}]: access denied 2023-07-10T17:09:27.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:26 smithi116 bash[28351]: audit 2023-07-10T17:09:26.600914+0000 mon.a (mon.0) 810 : audit [DBG] from='client.? 172.21.15.116:0/1800160153' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:09:28.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:28 smithi192 bash[23730]: cluster 2023-07-10T17:09:26.614282+0000 mgr.y (mgr.15252) 58 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:28.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:28 smithi116 bash[23740]: cluster 2023-07-10T17:09:26.614282+0000 mgr.y (mgr.15252) 58 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:28.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:28 smithi116 bash[28351]: cluster 2023-07-10T17:09:26.614282+0000 mgr.y (mgr.15252) 58 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:29.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:29 smithi192 bash[23730]: cluster 2023-07-10T17:09:28.614958+0000 mgr.y (mgr.15252) 59 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:29.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:29 smithi116 bash[23740]: cluster 2023-07-10T17:09:28.614958+0000 mgr.y (mgr.15252) 59 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:29.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:29 smithi116 bash[28351]: cluster 2023-07-10T17:09:28.614958+0000 mgr.y (mgr.15252) 59 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:30.280 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:29 smithi116 systemd[1]: Stopping Ceph mon.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:09:30.280 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:30 smithi116 bash[23740]: debug 2023-07-10T17:09:30.034+0000 7f30c1664700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-07-10T17:09:30.280 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:30 smithi116 bash[23740]: debug 2023-07-10T17:09:30.034+0000 7f30c1664700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-10T17:09:31.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:30 smithi116 bash[63742]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-mon.a 2023-07-10T17:09:31.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:30 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.a.service: Succeeded. 2023-07-10T17:09:31.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:30 smithi116 systemd[1]: Stopped Ceph mon.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:09:31.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:31 smithi116 systemd[1]: Started Ceph mon.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:09:32.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.410+0000 7f058d4e7c80 0 set uid:gid to 167:167 (ceph:ceph) 2023-07-10T17:09:32.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.410+0000 7f058d4e7c80 0 ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc), process ceph-mon, pid 7 2023-07-10T17:09:32.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.410+0000 7f058d4e7c80 0 pidfile_write: ignore empty --pid-file 2023-07-10T17:09:32.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 0 load: jerasure load: lrc 2023-07-10T17:09:32.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: RocksDB version: 7.9.2 2023-07-10T17:09:32.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Git sha 0 2023-07-10T17:09:32.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Compile date 2023-07-07 23:11:19 2023-07-10T17:09:32.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: DB SUMMARY 2023-07-10T17:09:32.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: DB Session ID: 6W4IKWO3KYETHKEE4QV0 2023-07-10T17:09:32.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: CURRENT file: CURRENT 2023-07-10T17:09:32.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: IDENTITY file: IDENTITY 2023-07-10T17:09:32.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: MANIFEST file: MANIFEST-000008 size: 959 Bytes 2023-07-10T17:09:32.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000023.sst 2023-07-10T17:09:32.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000021.log size: 10283633 ; 2023-07-10T17:09:32.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.error_if_exists: 0 2023-07-10T17:09:32.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.create_if_missing: 0 2023-07-10T17:09:32.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.paranoid_checks: 1 2023-07-10T17:09:32.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2023-07-10T17:09:32.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-10T17:09:32.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-10T17:09:32.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.env: 0x56169b919720 2023-07-10T17:09:32.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.fs: PosixFileSystem 2023-07-10T17:09:32.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.info_log: 0x56169cef53e0 2023-07-10T17:09:32.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.max_file_opening_threads: 16 2023-07-10T17:09:32.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.statistics: (nil) 2023-07-10T17:09:32.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.use_fsync: 0 2023-07-10T17:09:32.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.max_log_file_size: 0 2023-07-10T17:09:32.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-10T17:09:32.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.log_file_time_to_roll: 0 2023-07-10T17:09:32.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.keep_log_file_num: 1000 2023-07-10T17:09:32.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.recycle_log_file_num: 0 2023-07-10T17:09:32.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.allow_fallocate: 1 2023-07-10T17:09:32.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.allow_mmap_reads: 0 2023-07-10T17:09:32.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.allow_mmap_writes: 0 2023-07-10T17:09:32.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.use_direct_reads: 0 2023-07-10T17:09:32.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-10T17:09:32.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.create_missing_column_families: 0 2023-07-10T17:09:32.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.db_log_dir: 2023-07-10T17:09:32.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.wal_dir: 2023-07-10T17:09:32.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.table_cache_numshardbits: 6 2023-07-10T17:09:32.651 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-07-10T17:09:32.651 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-07-10T17:09:32.651 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-10T17:09:32.651 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-10T17:09:32.651 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-07-10T17:09:32.651 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.advise_random_on_open: 1 2023-07-10T17:09:32.652 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.db_write_buffer_size: 0 2023-07-10T17:09:32.652 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.write_buffer_manager: 0x56169c182c80 2023-07-10T17:09:32.652 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-10T17:09:32.652 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-10T17:09:32.652 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.use_adaptive_mutex: 0 2023-07-10T17:09:32.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.rate_limiter: (nil) 2023-07-10T17:09:32.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-10T17:09:32.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.wal_recovery_mode: 2 2023-07-10T17:09:32.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.enable_thread_tracking: 0 2023-07-10T17:09:32.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.enable_pipelined_write: 0 2023-07-10T17:09:32.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.unordered_write: 0 2023-07-10T17:09:32.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-10T17:09:32.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-10T17:09:32.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-10T17:09:32.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-10T17:09:32.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.row_cache: None 2023-07-10T17:09:32.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.wal_filter: None 2023-07-10T17:09:32.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-10T17:09:32.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.allow_ingest_behind: 0 2023-07-10T17:09:32.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.two_write_queues: 0 2023-07-10T17:09:32.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.manual_wal_flush: 0 2023-07-10T17:09:32.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.wal_compression: 0 2023-07-10T17:09:32.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.atomic_flush: 0 2023-07-10T17:09:32.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-10T17:09:32.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.persist_stats_to_disk: 0 2023-07-10T17:09:32.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-07-10T17:09:32.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.log_readahead_size: 0 2023-07-10T17:09:32.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.414+0000 7f058d4e7c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-10T17:09:32.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.best_efforts_recovery: 0 2023-07-10T17:09:32.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-10T17:09:32.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-10T17:09:32.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.allow_data_in_errors: 0 2023-07-10T17:09:32.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.db_host_id: __hostname__ 2023-07-10T17:09:32.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.enforce_single_del_contracts: true 2023-07-10T17:09:32.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_background_jobs: 2 2023-07-10T17:09:32.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_background_compactions: -1 2023-07-10T17:09:32.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_subcompactions: 1 2023-07-10T17:09:32.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-10T17:09:32.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-10T17:09:32.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.delayed_write_rate : 16777216 2023-07-10T17:09:32.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_total_wal_size: 0 2023-07-10T17:09:32.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-10T17:09:32.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.stats_dump_period_sec: 600 2023-07-10T17:09:32.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.stats_persist_period_sec: 600 2023-07-10T17:09:32.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-10T17:09:32.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_open_files: -1 2023-07-10T17:09:32.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.bytes_per_sync: 0 2023-07-10T17:09:32.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-07-10T17:09:32.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-07-10T17:09:32.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compaction_readahead_size: 0 2023-07-10T17:09:32.659 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_background_flushes: -1 2023-07-10T17:09:32.659 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Compression algorithms supported: 2023-07-10T17:09:32.659 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-10T17:09:32.659 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: kZSTD supported: 0 2023-07-10T17:09:32.659 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: kXpressCompression supported: 0 2023-07-10T17:09:32.659 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: kLZ4HCCompression supported: 1 2023-07-10T17:09:32.659 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: kZlibCompression supported: 1 2023-07-10T17:09:32.659 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: kSnappyCompression supported: 1 2023-07-10T17:09:32.660 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: kLZ4Compression supported: 1 2023-07-10T17:09:32.660 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: kBZip2Compression supported: 0 2023-07-10T17:09:32.660 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-07-10T17:09:32.660 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2023-07-10T17:09:32.660 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000008 2023-07-10T17:09:32.660 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-10T17:09:32.660 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-10T17:09:32.661 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.merge_operator: 2023-07-10T17:09:32.661 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compaction_filter: None 2023-07-10T17:09:32.661 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compaction_filter_factory: None 2023-07-10T17:09:32.661 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.sst_partitioner_factory: None 2023-07-10T17:09:32.661 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-07-10T17:09:32.661 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.table_factory: BlockBasedTable 2023-07-10T17:09:32.661 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56169cef5500) 2023-07-10T17:09:32.661 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: cache_index_and_filter_blocks: 1 2023-07-10T17:09:32.662 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-10T17:09:32.662 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-10T17:09:32.662 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: pin_top_level_index_and_filter: 1 2023-07-10T17:09:32.662 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: index_type: 0 2023-07-10T17:09:32.662 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: data_block_index_type: 0 2023-07-10T17:09:32.662 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: index_shortening: 1 2023-07-10T17:09:32.662 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: data_block_hash_table_util_ratio: 0.750000 2023-07-10T17:09:32.663 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: checksum: 4 2023-07-10T17:09:32.663 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: no_block_cache: 0 2023-07-10T17:09:32.663 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: block_cache: 0x56169c2051f0 2023-07-10T17:09:32.663 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: block_cache_name: BinnedLRUCache 2023-07-10T17:09:32.663 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: block_cache_options: 2023-07-10T17:09:32.663 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: capacity : 536870912 2023-07-10T17:09:32.663 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: num_shard_bits : 4 2023-07-10T17:09:32.663 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: strict_capacity_limit : 0 2023-07-10T17:09:32.664 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: high_pri_pool_ratio: 0.000 2023-07-10T17:09:32.664 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: block_cache_compressed: (nil) 2023-07-10T17:09:32.664 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: persistent_cache: (nil) 2023-07-10T17:09:32.664 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: block_size: 4096 2023-07-10T17:09:32.664 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: block_size_deviation: 10 2023-07-10T17:09:32.664 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: block_restart_interval: 16 2023-07-10T17:09:32.664 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: index_block_restart_interval: 1 2023-07-10T17:09:32.664 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: metadata_block_size: 4096 2023-07-10T17:09:32.665 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: partition_filters: 0 2023-07-10T17:09:32.665 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: use_delta_encoding: 1 2023-07-10T17:09:32.665 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: filter_policy: bloomfilter 2023-07-10T17:09:32.665 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: whole_key_filtering: 1 2023-07-10T17:09:32.665 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: verify_compression: 0 2023-07-10T17:09:32.665 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: read_amp_bytes_per_bit: 0 2023-07-10T17:09:32.665 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: format_version: 5 2023-07-10T17:09:32.666 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: enable_index_compression: 1 2023-07-10T17:09:32.666 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: block_align: 0 2023-07-10T17:09:32.666 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: max_auto_readahead_size: 262144 2023-07-10T17:09:32.666 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: prepopulate_block_cache: 0 2023-07-10T17:09:32.666 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: initial_auto_readahead_size: 8192 2023-07-10T17:09:32.666 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: num_file_reads_for_auto_readahead: 2 2023-07-10T17:09:32.666 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.write_buffer_size: 33554432 2023-07-10T17:09:32.666 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_write_buffer_number: 2 2023-07-10T17:09:32.667 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compression: NoCompression 2023-07-10T17:09:32.667 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.bottommost_compression: Disabled 2023-07-10T17:09:32.667 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.prefix_extractor: nullptr 2023-07-10T17:09:32.667 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-10T17:09:32.667 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.num_levels: 7 2023-07-10T17:09:32.667 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-10T17:09:32.667 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-10T17:09:32.667 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-10T17:09:32.668 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-10T17:09:32.668 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-10T17:09:32.668 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-10T17:09:32.668 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-10T17:09:32.668 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-10T17:09:32.668 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-10T17:09:32.668 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-10T17:09:32.669 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-10T17:09:32.669 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-10T17:09:32.669 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compression_opts.window_bits: -14 2023-07-10T17:09:32.669 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compression_opts.level: 32767 2023-07-10T17:09:32.669 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compression_opts.strategy: 0 2023-07-10T17:09:32.669 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-10T17:09:32.669 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-10T17:09:32.669 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-10T17:09:32.670 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-10T17:09:32.670 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compression_opts.enabled: false 2023-07-10T17:09:32.670 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-10T17:09:32.670 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-10T17:09:32.670 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-10T17:09:32.670 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-10T17:09:32.670 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.target_file_size_base: 67108864 2023-07-10T17:09:32.670 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.target_file_size_multiplier: 1 2023-07-10T17:09:32.671 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-10T17:09:32.671 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-10T17:09:32.671 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-10T17:09:32.671 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-10T17:09:32.671 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-10T17:09:32.671 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-10T17:09:32.671 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-10T17:09:32.671 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-10T17:09:32.672 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-10T17:09:32.672 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-10T17:09:32.672 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-10T17:09:32.672 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-10T17:09:32.672 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-10T17:09:32.672 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.arena_block_size: 1048576 2023-07-10T17:09:32.672 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-10T17:09:32.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-10T17:09:32.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.disable_auto_compactions: 0 2023-07-10T17:09:32.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-10T17:09:32.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-10T17:09:32.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-10T17:09:32.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-10T17:09:32.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-10T17:09:32.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-10T17:09:32.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-10T17:09:32.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-10T17:09:32.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-10T17:09:32.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-10T17:09:32.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.table_properties_collectors: 2023-07-10T17:09:32.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.inplace_update_support: 0 2023-07-10T17:09:32.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-07-10T17:09:32.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-10T17:09:32.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-10T17:09:32.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.memtable_huge_page_size: 0 2023-07-10T17:09:32.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.bloom_locality: 0 2023-07-10T17:09:32.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.max_successive_merges: 0 2023-07-10T17:09:32.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-07-10T17:09:32.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.paranoid_file_checks: 0 2023-07-10T17:09:32.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.force_consistency_checks: 1 2023-07-10T17:09:32.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.report_bg_io_stats: 0 2023-07-10T17:09:32.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.ttl: 2592000 2023-07-10T17:09:32.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-07-10T17:09:32.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-10T17:09:32.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-10T17:09:32.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.enable_blob_files: false 2023-07-10T17:09:32.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.min_blob_size: 0 2023-07-10T17:09:32.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.blob_file_size: 268435456 2023-07-10T17:09:32.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.blob_compression_type: NoCompression 2023-07-10T17:09:32.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-07-10T17:09:32.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-10T17:09:32.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-10T17:09:32.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-10T17:09:32.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.blob_file_starting_level: 0 2023-07-10T17:09:32.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-10T17:09:32.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f0582505700 3 rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 23.sst 2023-07-10T17:09:32.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-07-10T17:09:32.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 25, last_sequence is 11320, log_number is 21,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-10T17:09:32.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 21 2023-07-10T17:09:32.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5f0a6f00-8914-4ff7-8b48-f8cde6ba11dc 2023-07-10T17:09:32.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1689008972421031, "job": 1, "event": "recovery_started", "wal_files": [21]} 2023-07-10T17:09:32.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.418+0000 7f058d4e7c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #21 mode 2 2023-07-10T17:09:32.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.454+0000 7f058d4e7c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1689008972458231, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 26, "file_size": 10009108, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11321, "largest_seqno": 13746, "table_properties": {"data_size": 9995181, "index_size": 10538, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2373, "raw_key_size": 18210, "raw_average_key_size": 19, "raw_value_size": 9971409, "raw_average_value_size": 10619, "num_data_blocks": 461, "num_entries": 939, "num_filter_entries": 939, "num_deletions": 253, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689008972, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5f0a6f00-8914-4ff7-8b48-f8cde6ba11dc", "db_session_id": "6W4IKWO3KYETHKEE4QV0", "orig_file_number": 26, "seqno_to_time_mapping": "N/A"}} 2023-07-10T17:09:32.679 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.454+0000 7f058d4e7c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1689008972458356, "job": 1, "event": "recovery_finished"} 2023-07-10T17:09:32.679 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.454+0000 7f058d4e7c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 28 2023-07-10T17:09:32.679 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.454+0000 7f058d4e7c80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-07-10T17:09:32.680 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.458+0000 7f058d4e7c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000021.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-10T17:09:32.680 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.458+0000 7f058d4e7c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56169c2a4000 2023-07-10T17:09:32.680 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.458+0000 7f058d4e7c80 4 rocksdb: DB pointer 0x56169c292000 2023-07-10T17:09:32.680 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.458+0000 7f057bcf8700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-10T17:09:32.680 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.458+0000 7f057bcf8700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-10T17:09:32.680 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: ** DB Stats ** 2023-07-10T17:09:32.680 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Uptime(secs): 0.0 total, 0.0 interval 2023-07-10T17:09:32.681 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-10T17:09:32.681 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-10T17:09:32.681 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-10T17:09:32.681 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-10T17:09:32.681 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-10T17:09:32.681 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-10T17:09:32.681 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: ** Compaction Stats [default] ** 2023-07-10T17:09:32.681 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-10T17:09:32.682 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-10T17:09:32.682 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: L0 1/0 9.55 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 550.2 0.02 0.00 1 0.017 0 0 0.0 0.0 2023-07-10T17:09:32.682 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: L6 1/0 23.29 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2023-07-10T17:09:32.682 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Sum 2/0 32.84 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 550.2 0.02 0.00 1 0.017 0 0 0.0 0.0 2023-07-10T17:09:32.682 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 550.2 0.02 0.00 1 0.017 0 0 0.0 0.0 2023-07-10T17:09:32.682 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: ** Compaction Stats [default] ** 2023-07-10T17:09:32.682 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-10T17:09:32.683 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-10T17:09:32.683 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 550.2 0.02 0.00 1 0.017 0 0 0.0 0.0 2023-07-10T17:09:32.683 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-10T17:09:32.683 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Uptime(secs): 0.0 total, 0.0 interval 2023-07-10T17:09:32.683 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Flush(GB): cumulative 0.009, interval 0.009 2023-07-10T17:09:32.683 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-10T17:09:32.683 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: AddFile(Total Files): cumulative 0, interval 0 2023-07-10T17:09:32.683 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: AddFile(L0 Files): cumulative 0, interval 0 2023-07-10T17:09:32.684 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: AddFile(Keys): cumulative 0, interval 0 2023-07-10T17:09:32.684 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Cumulative compaction: 0.01 GB write, 222.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-10T17:09:32.684 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Interval compaction: 0.01 GB write, 222.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-10T17:09:32.684 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-10T17:09:32.684 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Block cache BinnedLRUCache@0x56169c2051f0#7 capacity: 512.00 MB usage: 14.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 2.2e-05 secs_since: 0 2023-07-10T17:09:32.684 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: Block cache entry stats(count,size,portion): FilterBlock(1,2.53 KB,0.000482798%) IndexBlock(1,12.08 KB,0.00230372%) Misc(1,0.00 KB,0%) 2023-07-10T17:09:32.684 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: ** File Read Latency Histogram By Level [default] ** 2023-07-10T17:09:32.684 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.462+0000 7f058d4e7c80 0 starting mon.a rank 0 at public addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] at bind addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T17:09:32.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.462+0000 7f058d4e7c80 1 mon.a@-1(???) e3 preinit fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T17:09:32.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.462+0000 7f058d4e7c80 0 mon.a@-1(???).mds e1 new map 2023-07-10T17:09:32.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.462+0000 7f058d4e7c80 0 mon.a@-1(???).mds e1 print_map 2023-07-10T17:09:32.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: e1 2023-07-10T17:09:32.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: enable_multiple, ever_enabled_multiple: 1,1 2023-07-10T17:09:32.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-10T17:09:32.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: legacy client fscid: -1 2023-07-10T17:09:32.686 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: 2023-07-10T17:09:32.686 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: No filesystems configured 2023-07-10T17:09:32.686 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.462+0000 7f058d4e7c80 0 mon.a@-1(???).osd e76 crush map has features 3314933000854323200, adjusting msgr requires 2023-07-10T17:09:32.686 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.462+0000 7f058d4e7c80 0 mon.a@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2023-07-10T17:09:32.686 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.462+0000 7f058d4e7c80 0 mon.a@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2023-07-10T17:09:32.686 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.462+0000 7f058d4e7c80 0 mon.a@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2023-07-10T17:09:32.686 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:09:32 smithi116 bash[63900]: debug 2023-07-10T17:09:32.462+0000 7f058d4e7c80 1 mon.a@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2023-07-10T17:09:33.636 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:33 smithi192 bash[47487]: ignoring --setuser ceph since I am not root 2023-07-10T17:09:33.637 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:33 smithi192 bash[47487]: ignoring --setgroup ceph since I am not root 2023-07-10T17:09:33.637 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:33 smithi192 bash[47487]: debug 2023-07-10T17:09:33.175+0000 7f6f287d4700 1 -- 172.21.15.192:0/1391650990 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56082fadb1e0 con 0x56083085a800 2023-07-10T17:09:33.637 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:33 smithi192 bash[47487]: debug 2023-07-10T17:09:33.331+0000 7f6f32504200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-10T17:09:33.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:33 smithi192 bash[23730]: cluster 2023-07-10T17:09:30.616805+0000 mgr.y (mgr.15252) 60 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 117 B/s rd, 0 op/s 2023-07-10T17:09:33.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:33 smithi192 bash[23730]: cluster 2023-07-10T17:09:33.084848+0000 mon.a (mon.0) 1 : cluster [INF] mon.a calling monitor election 2023-07-10T17:09:33.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:33 smithi192 bash[23730]: cluster 2023-07-10T17:09:33.088354+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-10T17:09:33.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:33 smithi192 bash[23730]: cluster 2023-07-10T17:09:33.101748+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0]} removed_ranks: {} 2023-07-10T17:09:33.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:33 smithi192 bash[23730]: cluster 2023-07-10T17:09:33.102538+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2023-07-10T17:09:33.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:33 smithi192 bash[23730]: cluster 2023-07-10T17:09:33.102581+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2023-07-10T17:09:33.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:33 smithi192 bash[23730]: cluster 2023-07-10T17:09:33.103986+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e26: y(active, since 60s), standbys: x 2023-07-10T17:09:33.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:33 smithi192 bash[23730]: cluster 2023-07-10T17:09:33.118143+0000 mon.a (mon.0) 7 : cluster [DBG] mgrmap e27: y(active, since 60s), standbys: x 2023-07-10T17:09:33.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:33 smithi192 bash[23730]: cluster 2023-07-10T17:09:33.118221+0000 mon.a (mon.0) 8 : cluster [WRN] Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2023-07-10T17:09:33.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:33 smithi192 bash[23730]: cluster 2023-07-10T17:09:33.118250+0000 mon.a (mon.0) 9 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2023-07-10T17:09:33.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:33 smithi192 bash[23730]: cluster 2023-07-10T17:09:33.118266+0000 mon.a (mon.0) 10 : cluster [WRN] daemon prometheus.a on smithi192 is in error state 2023-07-10T17:09:33.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:33 smithi192 bash[23730]: audit 2023-07-10T17:09:33.118397+0000 mon.a (mon.0) 11 : audit [INF] from='mgr.15252 ' entity='' 2023-07-10T17:09:33.643 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:33 smithi116 bash[59394]: ignoring --setuser ceph since I am not root 2023-07-10T17:09:33.643 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:33 smithi116 bash[59394]: ignoring --setgroup ceph since I am not root 2023-07-10T17:09:33.643 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:33 smithi116 bash[59394]: debug 2023-07-10T17:09:33.182+0000 7f6a1471d700 1 -- 172.21.15.116:0/4179701311 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56084fd631e0 con 0x560850ae2800 2023-07-10T17:09:33.643 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:33 smithi116 bash[59394]: debug 2023-07-10T17:09:33.338+0000 7f6a1e44d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-10T17:09:33.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:33 smithi116 bash[28351]: cluster 2023-07-10T17:09:30.616805+0000 mgr.y (mgr.15252) 60 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 117 B/s rd, 0 op/s 2023-07-10T17:09:33.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:33 smithi116 bash[28351]: cluster 2023-07-10T17:09:33.084848+0000 mon.a (mon.0) 1 : cluster [INF] mon.a calling monitor election 2023-07-10T17:09:33.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:33 smithi116 bash[28351]: cluster 2023-07-10T17:09:33.088354+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-10T17:09:33.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:33 smithi116 bash[28351]: cluster 2023-07-10T17:09:33.101748+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0]} removed_ranks: {} 2023-07-10T17:09:33.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:33 smithi116 bash[28351]: cluster 2023-07-10T17:09:33.102538+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2023-07-10T17:09:33.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:33 smithi116 bash[28351]: cluster 2023-07-10T17:09:33.102581+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2023-07-10T17:09:33.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:33 smithi116 bash[28351]: cluster 2023-07-10T17:09:33.103986+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e26: y(active, since 60s), standbys: x 2023-07-10T17:09:33.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:33 smithi116 bash[28351]: cluster 2023-07-10T17:09:33.118143+0000 mon.a (mon.0) 7 : cluster [DBG] mgrmap e27: y(active, since 60s), standbys: x 2023-07-10T17:09:33.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:33 smithi116 bash[28351]: cluster 2023-07-10T17:09:33.118221+0000 mon.a (mon.0) 8 : cluster [WRN] Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2023-07-10T17:09:33.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:33 smithi116 bash[28351]: cluster 2023-07-10T17:09:33.118250+0000 mon.a (mon.0) 9 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2023-07-10T17:09:33.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:33 smithi116 bash[28351]: cluster 2023-07-10T17:09:33.118266+0000 mon.a (mon.0) 10 : cluster [WRN] daemon prometheus.a on smithi192 is in error state 2023-07-10T17:09:33.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:33 smithi116 bash[28351]: audit 2023-07-10T17:09:33.118397+0000 mon.a (mon.0) 11 : audit [INF] from='mgr.15252 ' entity='' 2023-07-10T17:09:34.636 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:34 smithi192 bash[47487]: debug 2023-07-10T17:09:34.187+0000 7f6f32504200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-10T17:09:34.637 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:34 smithi192 bash[47487]: debug 2023-07-10T17:09:34.287+0000 7f6f32504200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-10T17:09:34.637 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:34 smithi192 bash[47487]: debug 2023-07-10T17:09:34.387+0000 7f6f32504200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-10T17:09:34.643 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:34 smithi116 bash[59394]: debug 2023-07-10T17:09:34.210+0000 7f6a1e44d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-10T17:09:34.643 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:34 smithi116 bash[59394]: debug 2023-07-10T17:09:34.310+0000 7f6a1e44d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-10T17:09:34.643 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:34 smithi116 bash[59394]: debug 2023-07-10T17:09:34.410+0000 7f6a1e44d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-10T17:09:35.003 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:34 smithi192 bash[47487]: debug 2023-07-10T17:09:34.647+0000 7f6f32504200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-10T17:09:35.018 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:34 smithi116 bash[59394]: debug 2023-07-10T17:09:34.666+0000 7f6a1e44d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-10T17:09:35.373 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:35 smithi192 bash[23730]: cluster 2023-07-10T17:09:34.124910+0000 mon.a (mon.0) 12 : cluster [DBG] mgrmap e28: y(active, since 61s), standbys: x 2023-07-10T17:09:35.373 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:35 smithi192 bash[47487]: debug 2023-07-10T17:09:34.995+0000 7f6f32504200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-10T17:09:35.373 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:35 smithi192 bash[47487]: debug 2023-07-10T17:09:35.155+0000 7f6f32504200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-10T17:09:35.386 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:35 smithi116 bash[28351]: cluster 2023-07-10T17:09:34.124910+0000 mon.a (mon.0) 12 : cluster [DBG] mgrmap e28: y(active, since 61s), standbys: x 2023-07-10T17:09:35.387 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:35 smithi116 bash[59394]: debug 2023-07-10T17:09:35.010+0000 7f6a1e44d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-10T17:09:35.387 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:35 smithi116 bash[59394]: debug 2023-07-10T17:09:35.170+0000 7f6a1e44d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-10T17:09:35.636 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:35 smithi192 bash[47487]: debug 2023-07-10T17:09:35.367+0000 7f6f32504200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-10T17:09:35.643 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:35 smithi116 bash[59394]: debug 2023-07-10T17:09:35.382+0000 7f6a1e44d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-10T17:09:36.478 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:36 smithi192 bash[47487]: debug 2023-07-10T17:09:36.359+0000 7f6f32504200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-10T17:09:36.478 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:36 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 3. 2023-07-10T17:09:36.478 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:36 smithi192 systemd[1]: Stopped Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:09:36.478 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:36 smithi192 systemd[1]: Started Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:09:36.643 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:36 smithi116 bash[59394]: debug 2023-07-10T17:09:36.366+0000 7f6a1e44d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-10T17:09:36.643 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:36 smithi116 bash[59394]: debug 2023-07-10T17:09:36.474+0000 7f6a1e44d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-10T17:09:36.643 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:36 smithi116 bash[59394]: debug 2023-07-10T17:09:36.590+0000 7f6a1e44d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-10T17:09:36.886 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:36 smithi192 bash[47487]: debug 2023-07-10T17:09:36.471+0000 7f6f32504200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-10T17:09:36.887 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:36 smithi192 bash[47487]: debug 2023-07-10T17:09:36.587+0000 7f6f32504200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-10T17:09:37.382 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[47487]: debug 2023-07-10T17:09:37.067+0000 7f6f32504200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-10T17:09:37.383 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[47487]: debug 2023-07-10T17:09:37.291+0000 7f6f32504200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-10T17:09:37.393 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:37 smithi116 bash[59394]: debug 2023-07-10T17:09:37.066+0000 7f6a1e44d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-10T17:09:37.393 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:37 smithi116 bash[59394]: debug 2023-07-10T17:09:37.286+0000 7f6a1e44d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-10T17:09:37.899 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[47487]: debug 2023-07-10T17:09:37.691+0000 7f6f32504200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-10T17:09:37.899 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.458Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-10T17:09:37.900 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.458Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-10T17:09:37.900 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.458Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-10T17:09:37.900 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.458Z caller=main.go:339 host_details="(Linux 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 smithi192 (none))" 2023-07-10T17:09:37.900 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.458Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-10T17:09:37.900 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.458Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-10T17:09:37.900 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.462Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-10T17:09:37.901 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.462Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-10T17:09:37.901 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.467Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-10T17:09:37.901 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.467Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=10 2023-07-10T17:09:37.901 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.479Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=10 2023-07-10T17:09:37.901 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.479Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=10 2023-07-10T17:09:37.901 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.480Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=10 2023-07-10T17:09:37.901 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.480Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=10 2023-07-10T17:09:37.902 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.480Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=10 2023-07-10T17:09:37.902 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.481Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=10 2023-07-10T17:09:37.902 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.484Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=10 2023-07-10T17:09:37.902 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.484Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=10 2023-07-10T17:09:37.902 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.485Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=10 2023-07-10T17:09:37.902 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.485Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=10 2023-07-10T17:09:37.902 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.485Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=18.388654ms 2023-07-10T17:09:37.903 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.488Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-10T17:09:37.903 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.488Z caller=main.go:695 msg="TSDB started" 2023-07-10T17:09:37.903 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.488Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-10T17:09:37.903 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.488Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-10T17:09:37.903 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.488Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-10T17:09:37.903 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.488Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-10T17:09:37.903 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.488Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-10T17:09:37.904 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.488Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-10T17:09:37.904 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.488Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-10T17:09:37.904 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.488Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-10T17:09:37.904 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.488Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-10T17:09:37.904 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.489Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-10T17:09:37.904 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=info ts=2023-07-10T17:09:37.489Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-10T17:09:37.904 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:37 smithi192 bash[48763]: level=error ts=2023-07-10T17:09:37.489Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-10T17:09:38.143 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:37 smithi116 bash[59394]: debug 2023-07-10T17:09:37.658+0000 7f6a1e44d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-10T17:09:38.636 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:38 smithi192 bash[47487]: debug 2023-07-10T17:09:38.183+0000 7f6f32504200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-10T17:09:38.637 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:38 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-10T17:09:38.637 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:38 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-10T17:09:38.643 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:38 smithi116 bash[59394]: debug 2023-07-10T17:09:38.142+0000 7f6a1e44d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-10T17:09:39.136 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:38 smithi192 bash[47487]: debug 2023-07-10T17:09:38.735+0000 7f6f32504200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-10T17:09:39.137 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:38 smithi192 bash[47487]: debug 2023-07-10T17:09:38.843+0000 7f6f32504200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-10T17:09:39.140 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:38 smithi116 bash[59394]: debug 2023-07-10T17:09:38.682+0000 7f6a1e44d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-10T17:09:39.140 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:38 smithi116 bash[59394]: debug 2023-07-10T17:09:38.790+0000 7f6a1e44d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-10T17:09:39.393 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:39 smithi116 bash[59394]: debug 2023-07-10T17:09:39.134+0000 7f6a1e44d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-10T17:09:39.393 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:39 smithi116 bash[59394]: debug 2023-07-10T17:09:39.258+0000 7f6a1e44d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-10T17:09:39.449 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:39 smithi192 bash[47487]: debug 2023-07-10T17:09:39.191+0000 7f6f32504200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-10T17:09:39.450 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:39 smithi192 bash[47487]: debug 2023-07-10T17:09:39.299+0000 7f6f32504200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-10T17:09:39.523 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:39 smithi116 bash[59394]: debug 2023-07-10T17:09:39.402+0000 7f6a1e44d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-10T17:09:39.838 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:39 smithi116 bash[59394]: debug 2023-07-10T17:09:39.518+0000 7f6a1e44d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-10T17:09:39.839 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:39 smithi116 bash[59394]: debug 2023-07-10T17:09:39.642+0000 7f6a1e44d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-10T17:09:39.879 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:39 smithi192 bash[47487]: debug 2023-07-10T17:09:39.443+0000 7f6f32504200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-10T17:09:39.879 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:39 smithi192 bash[47487]: debug 2023-07-10T17:09:39.559+0000 7f6f32504200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-10T17:09:39.879 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:39 smithi192 bash[47487]: debug 2023-07-10T17:09:39.683+0000 7f6f32504200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-10T17:09:40.136 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:39 smithi192 bash[47487]: debug 2023-07-10T17:09:39.871+0000 7f6f32504200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-10T17:09:40.137 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:40 smithi192 bash[47487]: debug 2023-07-10T17:09:39.995+0000 7f6f32504200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-10T17:09:40.143 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:39 smithi116 bash[59394]: debug 2023-07-10T17:09:39.830+0000 7f6a1e44d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-10T17:09:40.143 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:39 smithi116 bash[59394]: debug 2023-07-10T17:09:39.954+0000 7f6a1e44d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-10T17:09:41.343 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: debug 2023-07-10T17:09:41.206+0000 7f25c3267700 -1 mon.c@1(peon).osd e77 update_from_paxos full map CRC mismatch, resetting to canonical 2023-07-10T17:09:41.343 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: cluster 2023-07-10T17:09:41.200848+0000 mon.a (mon.0) 13 : cluster [INF] Active manager daemon y restarted 2023-07-10T17:09:41.343 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: cluster 2023-07-10T17:09:41.201192+0000 mon.a (mon.0) 14 : cluster [INF] Activating manager daemon y 2023-07-10T17:09:41.343 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: cluster 2023-07-10T17:09:41.209098+0000 mon.a (mon.0) 15 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2023-07-10T17:09:41.344 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: cluster 2023-07-10T17:09:41.209491+0000 mon.a (mon.0) 16 : cluster [DBG] mgrmap e29: y(active, starting, since 0.0083833s), standbys: x 2023-07-10T17:09:41.344 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.226523+0000 mon.a (mon.0) 17 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:09:41.344 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.226746+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:09:41.344 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.226889+0000 mon.a (mon.0) 19 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:09:41.344 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.229586+0000 mon.a (mon.0) 20 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-10T17:09:41.344 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.229770+0000 mon.a (mon.0) 21 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-10T17:09:41.344 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.230022+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T17:09:41.345 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.230198+0000 mon.a (mon.0) 23 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T17:09:41.345 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.230370+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T17:09:41.345 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.230534+0000 mon.a (mon.0) 25 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T17:09:41.345 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.230701+0000 mon.a (mon.0) 26 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T17:09:41.345 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.230872+0000 mon.a (mon.0) 27 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T17:09:41.345 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.231024+0000 mon.a (mon.0) 28 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T17:09:41.345 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.231241+0000 mon.a (mon.0) 29 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T17:09:41.345 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.231582+0000 mon.a (mon.0) 30 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-10T17:09:41.346 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.231728+0000 mon.a (mon.0) 31 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-10T17:09:41.346 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.232131+0000 mon.a (mon.0) 32 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-10T17:09:41.346 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: cluster 2023-07-10T17:09:41.235342+0000 mon.a (mon.0) 33 : cluster [DBG] Standby manager daemon x restarted 2023-07-10T17:09:41.346 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: cluster 2023-07-10T17:09:41.235415+0000 mon.a (mon.0) 34 : cluster [DBG] Standby manager daemon x started 2023-07-10T17:09:41.346 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.236786+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.? 172.21.15.192:0/2894608533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-10T17:09:41.346 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.238626+0000 mon.b (mon.2) 86 : audit [DBG] from='mgr.? 172.21.15.192:0/2894608533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:09:41.346 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.239721+0000 mon.b (mon.2) 87 : audit [DBG] from='mgr.? 172.21.15.192:0/2894608533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-10T17:09:41.347 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: audit 2023-07-10T17:09:41.240101+0000 mon.b (mon.2) 88 : audit [DBG] from='mgr.? 172.21.15.192:0/2894608533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:09:41.347 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[28351]: cluster 2023-07-10T17:09:41.240959+0000 mon.a (mon.0) 35 : cluster [INF] Manager daemon y is now available 2023-07-10T17:09:41.347 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:41 smithi116 bash[59394]: debug 2023-07-10T17:09:41.194+0000 7f6a1e44d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-10T17:09:41.460 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[47487]: debug 2023-07-10T17:09:41.211+0000 7f6f32504200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-10T17:09:41.461 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: debug 2023-07-10T17:09:41.203+0000 7f172827a700 -1 mon.b@2(peon).osd e77 update_from_paxos full map CRC mismatch, resetting to canonical 2023-07-10T17:09:41.461 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: cluster 2023-07-10T17:09:41.200848+0000 mon.a (mon.0) 13 : cluster [INF] Active manager daemon y restarted 2023-07-10T17:09:41.461 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: cluster 2023-07-10T17:09:41.201192+0000 mon.a (mon.0) 14 : cluster [INF] Activating manager daemon y 2023-07-10T17:09:41.461 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: cluster 2023-07-10T17:09:41.209098+0000 mon.a (mon.0) 15 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2023-07-10T17:09:41.461 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: cluster 2023-07-10T17:09:41.209491+0000 mon.a (mon.0) 16 : cluster [DBG] mgrmap e29: y(active, starting, since 0.0083833s), standbys: x 2023-07-10T17:09:41.462 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.226523+0000 mon.a (mon.0) 17 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-10T17:09:41.462 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.226746+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-10T17:09:41.462 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.226889+0000 mon.a (mon.0) 19 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-10T17:09:41.462 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.229586+0000 mon.a (mon.0) 20 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-10T17:09:41.462 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.229770+0000 mon.a (mon.0) 21 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-10T17:09:41.462 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.230022+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T17:09:41.462 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.230198+0000 mon.a (mon.0) 23 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T17:09:41.463 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.230370+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T17:09:41.463 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.230534+0000 mon.a (mon.0) 25 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T17:09:41.463 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.230701+0000 mon.a (mon.0) 26 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T17:09:41.463 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.230872+0000 mon.a (mon.0) 27 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T17:09:41.463 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.231024+0000 mon.a (mon.0) 28 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T17:09:41.463 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.231241+0000 mon.a (mon.0) 29 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T17:09:41.463 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.231582+0000 mon.a (mon.0) 30 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-10T17:09:41.464 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.231728+0000 mon.a (mon.0) 31 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-10T17:09:41.464 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.232131+0000 mon.a (mon.0) 32 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-10T17:09:41.464 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: cluster 2023-07-10T17:09:41.235342+0000 mon.a (mon.0) 33 : cluster [DBG] Standby manager daemon x restarted 2023-07-10T17:09:41.464 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: cluster 2023-07-10T17:09:41.235415+0000 mon.a (mon.0) 34 : cluster [DBG] Standby manager daemon x started 2023-07-10T17:09:41.464 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.236786+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.? 172.21.15.192:0/2894608533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-10T17:09:41.464 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.238626+0000 mon.b (mon.2) 86 : audit [DBG] from='mgr.? 172.21.15.192:0/2894608533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-10T17:09:41.464 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.239721+0000 mon.b (mon.2) 87 : audit [DBG] from='mgr.? 172.21.15.192:0/2894608533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-10T17:09:41.465 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: audit 2023-07-10T17:09:41.240101+0000 mon.b (mon.2) 88 : audit [DBG] from='mgr.? 172.21.15.192:0/2894608533' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-10T17:09:41.465 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:41 smithi192 bash[23730]: cluster 2023-07-10T17:09:41.240959+0000 mon.a (mon.0) 35 : cluster [INF] Manager daemon y is now available 2023-07-10T17:09:42.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:42 smithi192 bash[23730]: cluster 2023-07-10T17:09:41.215841+0000 osd.0 (osd.0) 3 : cluster [WRN] failed to encode map e77 with expected crc 2023-07-10T17:09:42.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:42 smithi192 bash[23730]: cluster 2023-07-10T17:09:41.217666+0000 osd.5 (osd.5) 3 : cluster [WRN] failed to encode map e77 with expected crc 2023-07-10T17:09:42.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:42 smithi192 bash[23730]: audit 2023-07-10T17:09:41.277768+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:09:42.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:42 smithi192 bash[23730]: cluster 2023-07-10T17:09:41.298769+0000 osd.7 (osd.7) 3 : cluster [WRN] failed to encode map e77 with expected crc 2023-07-10T17:09:42.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:42 smithi192 bash[23730]: audit 2023-07-10T17:09:41.323228+0000 mon.a (mon.0) 37 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:09:42.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:42 smithi192 bash[23730]: cluster 2023-07-10T17:09:41.334381+0000 osd.3 (osd.3) 3 : cluster [WRN] failed to encode map e77 with expected crc 2023-07-10T17:09:42.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:42 smithi192 bash[23730]: cluster 2023-07-10T17:09:41.334961+0000 osd.4 (osd.4) 3 : cluster [WRN] failed to encode map e77 with expected crc 2023-07-10T17:09:42.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:42 smithi192 bash[23730]: audit 2023-07-10T17:09:41.350414+0000 mon.a (mon.0) 38 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:09:42.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:42 smithi192 bash[23730]: cephadm 2023-07-10T17:09:41.744716+0000 mgr.y (mgr.25390) 1 : cephadm [INF] [10/Jul/2023:17:09:41] ENGINE Bus STARTING 2023-07-10T17:09:42.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:42 smithi192 bash[23730]: audit 2023-07-10T17:09:41.830813+0000 mon.c (mon.1) 96 : audit [DBG] from='client.? 172.21.15.116:0/3737784949' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:09:42.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:42 smithi192 bash[23730]: cephadm 2023-07-10T17:09:41.847036+0000 mgr.y (mgr.25390) 2 : cephadm [INF] [10/Jul/2023:17:09:41] ENGINE Serving on http://172.21.15.116:8765 2023-07-10T17:09:42.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:42 smithi192 bash[23730]: cephadm 2023-07-10T17:09:41.974125+0000 mgr.y (mgr.25390) 3 : cephadm [INF] [10/Jul/2023:17:09:41] ENGINE Serving on https://172.21.15.116:7150 2023-07-10T17:09:42.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:42 smithi192 bash[23730]: cephadm 2023-07-10T17:09:41.974317+0000 mgr.y (mgr.25390) 4 : cephadm [INF] [10/Jul/2023:17:09:41] ENGINE Bus STARTED 2023-07-10T17:09:42.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:42 smithi192 bash[23730]: audit 2023-07-10T17:09:42.183498+0000 mon.a (mon.0) 39 : audit [INF] from='client.34155 172.21.15.116:0/2835375883' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]: access denied 2023-07-10T17:09:42.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:42 smithi192 bash[23730]: cluster 2023-07-10T17:09:42.217941+0000 mon.a (mon.0) 40 : cluster [DBG] mgrmap e30: y(active, since 1.01682s), standbys: x 2023-07-10T17:09:42.643 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:09:42 smithi116 bash[59394]: debug 2023-07-10T17:09:42.226+0000 7f69d6487700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-07-10T17:09:42.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:42 smithi116 bash[28351]: cluster 2023-07-10T17:09:41.215841+0000 osd.0 (osd.0) 3 : cluster [WRN] failed to encode map e77 with expected crc 2023-07-10T17:09:42.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:42 smithi116 bash[28351]: cluster 2023-07-10T17:09:41.217666+0000 osd.5 (osd.5) 3 : cluster [WRN] failed to encode map e77 with expected crc 2023-07-10T17:09:42.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:42 smithi116 bash[28351]: audit 2023-07-10T17:09:41.277768+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:09:42.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:42 smithi116 bash[28351]: cluster 2023-07-10T17:09:41.298769+0000 osd.7 (osd.7) 3 : cluster [WRN] failed to encode map e77 with expected crc 2023-07-10T17:09:42.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:42 smithi116 bash[28351]: audit 2023-07-10T17:09:41.323228+0000 mon.a (mon.0) 37 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-10T17:09:42.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:42 smithi116 bash[28351]: cluster 2023-07-10T17:09:41.334381+0000 osd.3 (osd.3) 3 : cluster [WRN] failed to encode map e77 with expected crc 2023-07-10T17:09:42.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:42 smithi116 bash[28351]: cluster 2023-07-10T17:09:41.334961+0000 osd.4 (osd.4) 3 : cluster [WRN] failed to encode map e77 with expected crc 2023-07-10T17:09:42.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:42 smithi116 bash[28351]: audit 2023-07-10T17:09:41.350414+0000 mon.a (mon.0) 38 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-10T17:09:42.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:42 smithi116 bash[28351]: cephadm 2023-07-10T17:09:41.744716+0000 mgr.y (mgr.25390) 1 : cephadm [INF] [10/Jul/2023:17:09:41] ENGINE Bus STARTING 2023-07-10T17:09:42.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:42 smithi116 bash[28351]: audit 2023-07-10T17:09:41.830813+0000 mon.c (mon.1) 96 : audit [DBG] from='client.? 172.21.15.116:0/3737784949' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:09:42.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:42 smithi116 bash[28351]: cephadm 2023-07-10T17:09:41.847036+0000 mgr.y (mgr.25390) 2 : cephadm [INF] [10/Jul/2023:17:09:41] ENGINE Serving on http://172.21.15.116:8765 2023-07-10T17:09:42.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:42 smithi116 bash[28351]: cephadm 2023-07-10T17:09:41.974125+0000 mgr.y (mgr.25390) 3 : cephadm [INF] [10/Jul/2023:17:09:41] ENGINE Serving on https://172.21.15.116:7150 2023-07-10T17:09:42.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:42 smithi116 bash[28351]: cephadm 2023-07-10T17:09:41.974317+0000 mgr.y (mgr.25390) 4 : cephadm [INF] [10/Jul/2023:17:09:41] ENGINE Bus STARTED 2023-07-10T17:09:42.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:42 smithi116 bash[28351]: audit 2023-07-10T17:09:42.183498+0000 mon.a (mon.0) 39 : audit [INF] from='client.34155 172.21.15.116:0/2835375883' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]: access denied 2023-07-10T17:09:42.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:42 smithi116 bash[28351]: cluster 2023-07-10T17:09:42.217941+0000 mon.a (mon.0) 40 : cluster [DBG] mgrmap e30: y(active, since 1.01682s), standbys: x 2023-07-10T17:09:43.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:43 smithi116 bash[28351]: cluster 2023-07-10T17:09:42.231725+0000 mgr.y (mgr.25390) 5 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:43.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:43 smithi116 bash[28351]: cluster 2023-07-10T17:09:43.210357+0000 mon.a (mon.0) 41 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2023-07-10T17:09:43.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:43 smithi116 bash[28351]: cluster 2023-07-10T17:09:43.210394+0000 mon.a (mon.0) 42 : cluster [INF] Cluster is now healthy 2023-07-10T17:09:43.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:43 smithi192 bash[23730]: cluster 2023-07-10T17:09:42.231725+0000 mgr.y (mgr.25390) 5 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:43.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:43 smithi192 bash[23730]: cluster 2023-07-10T17:09:43.210357+0000 mon.a (mon.0) 41 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2023-07-10T17:09:43.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:43 smithi192 bash[23730]: cluster 2023-07-10T17:09:43.210394+0000 mon.a (mon.0) 42 : cluster [INF] Cluster is now healthy 2023-07-10T17:09:44.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:44 smithi192 bash[23730]: cluster 2023-07-10T17:09:43.231670+0000 mgr.y (mgr.25390) 6 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:44.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:44 smithi192 bash[23730]: cluster 2023-07-10T17:09:43.492770+0000 mon.a (mon.0) 43 : cluster [DBG] mgrmap e31: y(active, since 2s), standbys: x 2023-07-10T17:09:44.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:44 smithi116 bash[28351]: cluster 2023-07-10T17:09:43.231670+0000 mgr.y (mgr.25390) 6 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:44.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:44 smithi116 bash[28351]: cluster 2023-07-10T17:09:43.492770+0000 mon.a (mon.0) 43 : cluster [DBG] mgrmap e31: y(active, since 2s), standbys: x 2023-07-10T17:09:46.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:46 smithi192 bash[23730]: cluster 2023-07-10T17:09:45.232364+0000 mgr.y (mgr.25390) 7 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:46.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:46 smithi116 bash[28351]: cluster 2023-07-10T17:09:45.232364+0000 mgr.y (mgr.25390) 7 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:48.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:48 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 4. 2023-07-10T17:09:48.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:48 smithi192 systemd[1]: Stopped Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:09:48.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:48 smithi192 systemd[1]: Started Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:09:48.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:48 smithi192 bash[23730]: cluster 2023-07-10T17:09:47.233995+0000 mgr.y (mgr.25390) 8 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:48.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:48 smithi192 bash[23730]: audit 2023-07-10T17:09:48.304040+0000 mon.a (mon.0) 44 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:48.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:48 smithi192 bash[23730]: audit 2023-07-10T17:09:48.313237+0000 mon.a (mon.0) 45 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:48.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:48 smithi192 bash[23730]: audit 2023-07-10T17:09:48.410314+0000 mon.a (mon.0) 46 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:48.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:48 smithi192 bash[23730]: audit 2023-07-10T17:09:48.422006+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:48.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:48 smithi116 bash[28351]: cluster 2023-07-10T17:09:47.233995+0000 mgr.y (mgr.25390) 8 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:48.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:48 smithi116 bash[28351]: audit 2023-07-10T17:09:48.304040+0000 mon.a (mon.0) 44 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:48.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:48 smithi116 bash[28351]: audit 2023-07-10T17:09:48.313237+0000 mon.a (mon.0) 45 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:48.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:48 smithi116 bash[28351]: audit 2023-07-10T17:09:48.410314+0000 mon.a (mon.0) 46 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:48.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:48 smithi116 bash[28351]: audit 2023-07-10T17:09:48.422006+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:50.058 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.844Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-10T17:09:50.058 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.844Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-10T17:09:50.058 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.844Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-10T17:09:50.059 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.844Z caller=main.go:339 host_details="(Linux 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 smithi192 (none))" 2023-07-10T17:09:50.059 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.844Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-10T17:09:50.059 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.844Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-10T17:09:50.059 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.846Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-10T17:09:50.059 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.846Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-10T17:09:50.059 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.849Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-10T17:09:50.059 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.850Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=11 2023-07-10T17:09:50.060 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.863Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=11 2023-07-10T17:09:50.060 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.864Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=11 2023-07-10T17:09:50.060 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.864Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=11 2023-07-10T17:09:50.060 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.864Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=11 2023-07-10T17:09:50.060 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.864Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=11 2023-07-10T17:09:50.060 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.865Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=11 2023-07-10T17:09:50.061 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.865Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=11 2023-07-10T17:09:50.061 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.865Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=11 2023-07-10T17:09:50.061 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.866Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=11 2023-07-10T17:09:50.061 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.868Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=11 2023-07-10T17:09:50.061 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.868Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=11 2023-07-10T17:09:50.061 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.868Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=18.815922ms 2023-07-10T17:09:50.062 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.871Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-10T17:09:50.062 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.871Z caller=main.go:695 msg="TSDB started" 2023-07-10T17:09:50.062 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.871Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-10T17:09:50.062 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.872Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-10T17:09:50.062 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.872Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-10T17:09:50.062 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.872Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-10T17:09:50.062 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.872Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-10T17:09:50.063 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.872Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-10T17:09:50.063 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.872Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-10T17:09:50.063 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.872Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-10T17:09:50.063 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.872Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-10T17:09:50.063 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.872Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-10T17:09:50.063 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=info ts=2023-07-10T17:09:49.872Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-10T17:09:50.063 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:49 smithi192 bash[49492]: level=error ts=2023-07-10T17:09:49.872Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-10T17:09:50.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:50 smithi192 bash[23730]: audit 2023-07-10T17:09:49.178996+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:50.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:50 smithi192 bash[23730]: audit 2023-07-10T17:09:49.187809+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:50.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:50 smithi192 bash[23730]: audit 2023-07-10T17:09:49.192422+0000 mon.a (mon.0) 50 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:50.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:50 smithi192 bash[23730]: audit 2023-07-10T17:09:49.198955+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:50.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:50 smithi116 bash[28351]: audit 2023-07-10T17:09:49.178996+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:50.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:50 smithi116 bash[28351]: audit 2023-07-10T17:09:49.187809+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:50.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:50 smithi116 bash[28351]: audit 2023-07-10T17:09:49.192422+0000 mon.a (mon.0) 50 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:50.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:50 smithi116 bash[28351]: audit 2023-07-10T17:09:49.198955+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:50.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:50 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-10T17:09:50.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:09:50 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-10T17:09:51.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:51 smithi192 bash[23730]: cluster 2023-07-10T17:09:49.234683+0000 mgr.y (mgr.25390) 9 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:51.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:51 smithi116 bash[28351]: cluster 2023-07-10T17:09:49.234683+0000 mgr.y (mgr.25390) 9 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:53.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:53 smithi192 bash[23730]: cluster 2023-07-10T17:09:51.235421+0000 mgr.y (mgr.25390) 10 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:53.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:53 smithi116 bash[28351]: cluster 2023-07-10T17:09:51.235421+0000 mgr.y (mgr.25390) 10 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:55.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:55 smithi192 bash[23730]: cluster 2023-07-10T17:09:53.237029+0000 mgr.y (mgr.25390) 11 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:55.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:55 smithi116 bash[28351]: cluster 2023-07-10T17:09:53.237029+0000 mgr.y (mgr.25390) 11 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:56.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:56 smithi192 bash[23730]: cluster 2023-07-10T17:09:55.237771+0000 mgr.y (mgr.25390) 12 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:56.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:56 smithi116 bash[28351]: cluster 2023-07-10T17:09:55.237771+0000 mgr.y (mgr.25390) 12 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:09:57.000 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:09:57.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:57 smithi116 bash[28351]: audit 2023-07-10T17:09:56.426782+0000 mon.c (mon.1) 97 : audit [DBG] from='client.? 172.21.15.116:0/2649887320' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:09:57.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:57 smithi116 bash[28351]: audit 2023-07-10T17:09:56.797829+0000 mon.c (mon.1) 98 : audit [INF] from='client.? 172.21.15.116:0/1164294302' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]: access denied 2023-07-10T17:09:57.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:57 smithi116 bash[28351]: audit 2023-07-10T17:09:56.981724+0000 mgr.y (mgr.25390) 13 : audit [DBG] from='client.25465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:57.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:57 smithi192 bash[23730]: audit 2023-07-10T17:09:56.426782+0000 mon.c (mon.1) 97 : audit [DBG] from='client.? 172.21.15.116:0/2649887320' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:09:57.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:57 smithi192 bash[23730]: audit 2023-07-10T17:09:56.797829+0000 mon.c (mon.1) 98 : audit [INF] from='client.? 172.21.15.116:0/1164294302' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]: access denied 2023-07-10T17:09:57.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:57 smithi192 bash[23730]: audit 2023-07-10T17:09:56.981724+0000 mgr.y (mgr.25390) 13 : audit [DBG] from='client.25465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:57.772 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:09:57.772 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (3m) 9s ago 9m 10.7M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:09:57.772 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (108s) 9s ago 9m 25.4M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:09:57.772 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 error 9s ago 6m - - 2023-07-10T17:09:57.773 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (52s) 9s ago 13m 426M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:09:57.773 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (102s) 9s ago 15m 457M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:09:57.773 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (26s) 9s ago 15m 38.2M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:09:57.773 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (14m) 9s ago 14m 69.1M 2048M 16.2.0 24ecd6d5f14c 9ad4329e5488 2023-07-10T17:09:57.773 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (14m) 9s ago 14m 66.0M 2048M 16.2.0 24ecd6d5f14c 31e3ac12741b 2023-07-10T17:09:57.773 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (3m) 9s ago 9m 9247k - 0.18.1 e5a616e4b9cf 3bd058d91353 2023-07-10T17:09:57.773 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (2m) 9s ago 9m 3563k - 0.18.1 e5a616e4b9cf 63551615c679 2023-07-10T17:09:57.773 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (13m) 9s ago 13m 55.4M 4096M 16.2.0 24ecd6d5f14c 4fa26367c830 2023-07-10T17:09:57.774 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (12m) 9s ago 12m 53.5M 4096M 16.2.0 24ecd6d5f14c c157509b3fd4 2023-07-10T17:09:57.774 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (12m) 9s ago 12m 49.5M 4096M 16.2.0 24ecd6d5f14c f1fd77b3c454 2023-07-10T17:09:57.774 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (11m) 9s ago 12m 54.5M 4096M 16.2.0 24ecd6d5f14c 60949d479c11 2023-07-10T17:09:57.774 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (11m) 9s ago 11m 54.5M 4096M 16.2.0 24ecd6d5f14c 75fc0b36aab8 2023-07-10T17:09:57.774 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (11m) 9s ago 11m 52.4M 4096M 16.2.0 24ecd6d5f14c d673065333dd 2023-07-10T17:09:57.775 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (10m) 9s ago 10m 53.3M 4096M 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:09:57.775 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (10m) 9s ago 10m 56.7M 4096M 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:09:57.775 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 error 9s ago 9m - - 2023-07-10T17:09:57.775 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (7m) 9s ago 7m 54.5M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:09:57.775 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (7m) 9s ago 7m 54.3M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:09:57.775 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (7m) 9s ago 7m 54.4M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:09:57.775 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (7m) 9s ago 7m 56.2M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:09:58.205 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:09:58.205 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:09:58.205 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2, 2023-07-10T17:09:58.206 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 1 2023-07-10T17:09:58.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:09:58.206 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:09:58.206 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:09:58.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:09:58.206 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:09:58.206 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-10T17:09:58.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:09:58.207 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:09:58.207 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:09:58.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:09:58.207 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:09:58.207 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 14, 2023-07-10T17:09:58.207 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:09:58.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:09:58.208 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:09:58.578 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:09:58.578 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:09:58.578 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:09:58.578 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:09:58.579 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:09:58.579 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2023-07-10T17:09:58.579 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:09:58.579 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "3/23 daemons upgraded", 2023-07-10T17:09:58.579 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "", 2023-07-10T17:09:58.579 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:09:58.579 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:09:58.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:58 smithi116 bash[28351]: cluster 2023-07-10T17:09:57.239327+0000 mgr.y (mgr.25390) 14 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:58.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:58 smithi116 bash[28351]: audit 2023-07-10T17:09:57.363273+0000 mgr.y (mgr.25390) 15 : audit [DBG] from='client.25471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:58.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:58 smithi116 bash[28351]: audit 2023-07-10T17:09:57.744423+0000 mgr.y (mgr.25390) 16 : audit [DBG] from='client.25477 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:58.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:58 smithi116 bash[28351]: audit 2023-07-10T17:09:58.201338+0000 mon.a (mon.0) 52 : audit [DBG] from='client.34203 172.21.15.116:0/1768468329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:09:58.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:58 smithi192 bash[23730]: cluster 2023-07-10T17:09:57.239327+0000 mgr.y (mgr.25390) 14 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:09:58.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:58 smithi192 bash[23730]: audit 2023-07-10T17:09:57.363273+0000 mgr.y (mgr.25390) 15 : audit [DBG] from='client.25471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:58.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:58 smithi192 bash[23730]: audit 2023-07-10T17:09:57.744423+0000 mgr.y (mgr.25390) 16 : audit [DBG] from='client.25477 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:58.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:58 smithi192 bash[23730]: audit 2023-07-10T17:09:58.201338+0000 mon.a (mon.0) 52 : audit [DBG] from='client.34203 172.21.15.116:0/1768468329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:09:59.011 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2023-07-10T17:09:59.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:59 smithi192 bash[23730]: audit 2023-07-10T17:09:58.573745+0000 mgr.y (mgr.25390) 17 : audit [DBG] from='client.25489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:59.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:59 smithi192 bash[23730]: audit 2023-07-10T17:09:59.008117+0000 mon.c (mon.1) 99 : audit [DBG] from='client.? 172.21.15.116:0/3069563215' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:09:59.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:59 smithi192 bash[23730]: audit 2023-07-10T17:09:59.098231+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:59.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:59 smithi192 bash[23730]: audit 2023-07-10T17:09:59.108591+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:59.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:59 smithi192 bash[23730]: audit 2023-07-10T17:09:59.111286+0000 mon.a (mon.0) 55 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:09:59.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:59 smithi192 bash[23730]: audit 2023-07-10T17:09:59.112529+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:09:59.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:59 smithi192 bash[23730]: cephadm 2023-07-10T17:09:59.113529+0000 mgr.y (mgr.25390) 18 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:09:59.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:59 smithi192 bash[23730]: cluster 2023-07-10T17:09:59.114638+0000 mgr.y (mgr.25390) 19 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:09:59.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:59 smithi192 bash[23730]: audit 2023-07-10T17:09:59.121473+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:59.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:59 smithi192 bash[23730]: audit 2023-07-10T17:09:59.208871+0000 mon.a (mon.0) 58 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:09:59.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:59 smithi192 bash[23730]: audit 2023-07-10T17:09:59.214394+0000 mon.a (mon.0) 59 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:09:59.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:59 smithi192 bash[23730]: audit 2023-07-10T17:09:59.216783+0000 mon.a (mon.0) 60 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-10T17:09:59.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:09:59 smithi192 bash[23730]: audit 2023-07-10T17:09:59.218328+0000 mon.a (mon.0) 61 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2023-07-10T17:09:59.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:59 smithi116 bash[28351]: audit 2023-07-10T17:09:58.573745+0000 mgr.y (mgr.25390) 17 : audit [DBG] from='client.25489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:09:59.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:59 smithi116 bash[28351]: audit 2023-07-10T17:09:59.008117+0000 mon.c (mon.1) 99 : audit [DBG] from='client.? 172.21.15.116:0/3069563215' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:09:59.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:59 smithi116 bash[28351]: audit 2023-07-10T17:09:59.098231+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:59.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:59 smithi116 bash[28351]: audit 2023-07-10T17:09:59.108591+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:59.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:59 smithi116 bash[28351]: audit 2023-07-10T17:09:59.111286+0000 mon.a (mon.0) 55 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:09:59.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:59 smithi116 bash[28351]: audit 2023-07-10T17:09:59.112529+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:09:59.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:59 smithi116 bash[28351]: cephadm 2023-07-10T17:09:59.113529+0000 mgr.y (mgr.25390) 18 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:09:59.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:59 smithi116 bash[28351]: cluster 2023-07-10T17:09:59.114638+0000 mgr.y (mgr.25390) 19 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:09:59.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:59 smithi116 bash[28351]: audit 2023-07-10T17:09:59.121473+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:09:59.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:59 smithi116 bash[28351]: audit 2023-07-10T17:09:59.208871+0000 mon.a (mon.0) 58 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:09:59.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:59 smithi116 bash[28351]: audit 2023-07-10T17:09:59.214394+0000 mon.a (mon.0) 59 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:09:59.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:59 smithi116 bash[28351]: audit 2023-07-10T17:09:59.216783+0000 mon.a (mon.0) 60 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-10T17:09:59.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:09:59 smithi116 bash[28351]: audit 2023-07-10T17:09:59.218328+0000 mon.a (mon.0) 61 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2023-07-10T17:10:00.850 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:00 smithi116 bash[28351]: cephadm 2023-07-10T17:09:59.219379+0000 mgr.y (mgr.25390) 20 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2023-07-10T17:10:00.850 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:00 smithi116 bash[28351]: cluster 2023-07-10T17:10:00.000194+0000 mon.a (mon.0) 62 : cluster [INF] overall HEALTH_OK 2023-07-10T17:10:00.850 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:00 smithi116 bash[28351]: cluster 2023-07-10T17:10:00.105153+0000 mon.a (mon.0) 63 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:10:00.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:00 smithi192 bash[23730]: cephadm 2023-07-10T17:09:59.219379+0000 mgr.y (mgr.25390) 20 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2023-07-10T17:10:00.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:00 smithi192 bash[23730]: cluster 2023-07-10T17:10:00.000194+0000 mon.a (mon.0) 62 : cluster [INF] overall HEALTH_OK 2023-07-10T17:10:00.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:00 smithi192 bash[23730]: cluster 2023-07-10T17:10:00.105153+0000 mon.a (mon.0) 63 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:10:00.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:00 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 5. 2023-07-10T17:10:00.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:00 smithi192 systemd[1]: Stopped Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:10:00.888 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:00 smithi192 systemd[1]: Started Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:10:01.825 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[23730]: cluster 2023-07-10T17:10:01.115429+0000 mgr.y (mgr.25390) 21 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:01.825 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[23730]: audit 2023-07-10T17:10:01.448292+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:01.825 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[23730]: audit 2023-07-10T17:10:01.450164+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:10:01.825 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[23730]: audit 2023-07-10T17:10:01.452396+0000 mon.a (mon.0) 66 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:10:01.825 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[23730]: audit 2023-07-10T17:10:01.454067+0000 mon.a (mon.0) 67 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:01.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:01 smithi116 bash[28351]: cluster 2023-07-10T17:10:01.115429+0000 mgr.y (mgr.25390) 21 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:01.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:01 smithi116 bash[28351]: audit 2023-07-10T17:10:01.448292+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:01.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:01 smithi116 bash[28351]: audit 2023-07-10T17:10:01.450164+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:10:01.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:01 smithi116 bash[28351]: audit 2023-07-10T17:10:01.452396+0000 mon.a (mon.0) 66 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:10:01.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:01 smithi116 bash[28351]: audit 2023-07-10T17:10:01.454067+0000 mon.a (mon.0) 67 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:02.137 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.821Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2023-07-10T17:10:02.137 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.821Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2023-07-10T17:10:02.138 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.821Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2023-07-10T17:10:02.138 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.822Z caller=main.go:339 host_details="(Linux 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 smithi192 (none))" 2023-07-10T17:10:02.138 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.822Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2023-07-10T17:10:02.138 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.822Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-10T17:10:02.138 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.823Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2023-07-10T17:10:02.138 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.823Z caller=main.go:678 msg="Starting TSDB ..." 2023-07-10T17:10:02.138 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.828Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2023-07-10T17:10:02.139 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.829Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=12 2023-07-10T17:10:02.139 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.847Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=12 2023-07-10T17:10:02.139 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.847Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=12 2023-07-10T17:10:02.139 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.847Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=12 2023-07-10T17:10:02.139 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.847Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=12 2023-07-10T17:10:02.139 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.848Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=12 2023-07-10T17:10:02.139 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.848Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=12 2023-07-10T17:10:02.139 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.848Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=12 2023-07-10T17:10:02.140 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.848Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=12 2023-07-10T17:10:02.140 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.848Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=12 2023-07-10T17:10:02.140 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.849Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=12 2023-07-10T17:10:02.140 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.851Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=12 2023-07-10T17:10:02.140 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.851Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=12 maxSegment=12 2023-07-10T17:10:02.140 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.851Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=22.634842ms 2023-07-10T17:10:02.140 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.854Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2023-07-10T17:10:02.141 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.854Z caller=main.go:695 msg="TSDB started" 2023-07-10T17:10:02.141 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.854Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-10T17:10:02.141 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.855Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2023-07-10T17:10:02.141 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.855Z caller=main.go:561 msg="Stopping notify discovery manager..." 2023-07-10T17:10:02.141 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.855Z caller=main.go:583 msg="Stopping scrape manager..." 2023-07-10T17:10:02.141 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.855Z caller=main.go:557 msg="Notify discovery manager stopped" 2023-07-10T17:10:02.141 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.855Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2023-07-10T17:10:02.142 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.855Z caller=main.go:543 msg="Scrape discovery manager stopped" 2023-07-10T17:10:02.142 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.855Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2023-07-10T17:10:02.142 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.855Z caller=main.go:577 msg="Scrape manager stopped" 2023-07-10T17:10:02.142 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.855Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2023-07-10T17:10:02.142 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=info ts=2023-07-10T17:10:01.855Z caller=main.go:749 msg="Notifier manager stopped" 2023-07-10T17:10:02.142 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:01 smithi192 bash[49879]: level=error ts=2023-07-10T17:10:01.855Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 11: field http_sd_configs not found in type config.plain\n line 17: field http_sd_configs not found in type config.plain\n line 21: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain" 2023-07-10T17:10:02.804 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:02 smithi116 bash[28351]: cephadm 2023-07-10T17:10:01.438560+0000 mgr.y (mgr.25390) 22 : cephadm [INF] Upgrade: Updating mon.c 2023-07-10T17:10:02.804 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:02 smithi116 bash[28351]: cephadm 2023-07-10T17:10:01.455702+0000 mgr.y (mgr.25390) 23 : cephadm [INF] Deploying daemon mon.c on smithi116 2023-07-10T17:10:02.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:02 smithi192 bash[23730]: cephadm 2023-07-10T17:10:01.438560+0000 mgr.y (mgr.25390) 22 : cephadm [INF] Upgrade: Updating mon.c 2023-07-10T17:10:02.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:02 smithi192 bash[23730]: cephadm 2023-07-10T17:10:01.455702+0000 mgr.y (mgr.25390) 23 : cephadm [INF] Deploying daemon mon.c on smithi116 2023-07-10T17:10:02.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:02 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2023-07-10T17:10:02.887 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:02 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-10T17:10:03.857 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:03 smithi116 bash[28351]: cluster 2023-07-10T17:10:03.117083+0000 mgr.y (mgr.25390) 24 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:03.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:03 smithi192 bash[23730]: cluster 2023-07-10T17:10:03.117083+0000 mgr.y (mgr.25390) 24 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:04.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:03 smithi116 systemd[1]: Stopping Ceph mon.c for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:10:04.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:03 smithi116 bash[28351]: debug 2023-07-10T17:10:03.965+0000 7f25c9a74700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-07-10T17:10:04.143 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:03 smithi116 bash[28351]: debug 2023-07-10T17:10:03.965+0000 7f25c9a74700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-07-10T17:10:05.376 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:05 smithi116 bash[65816]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-mon.c 2023-07-10T17:10:05.376 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:05 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.c.service: Succeeded. 2023-07-10T17:10:05.376 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:05 smithi116 systemd[1]: Stopped Ceph mon.c for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:10:05.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:05 smithi116 systemd[1]: Started Ceph mon.c for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:10:06.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.541+0000 7fd503378c80 0 set uid:gid to 167:167 (ceph:ceph) 2023-07-10T17:10:06.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.541+0000 7fd503378c80 0 ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc), process ceph-mon, pid 7 2023-07-10T17:10:06.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.541+0000 7fd503378c80 0 pidfile_write: ignore empty --pid-file 2023-07-10T17:10:06.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 0 load: jerasure load: lrc 2023-07-10T17:10:06.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: RocksDB version: 7.9.2 2023-07-10T17:10:06.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Git sha 0 2023-07-10T17:10:06.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Compile date 2023-07-07 23:11:19 2023-07-10T17:10:06.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: DB SUMMARY 2023-07-10T17:10:06.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: DB Session ID: 2QMPLLEXFH2M63G0EV4L 2023-07-10T17:10:06.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: CURRENT file: CURRENT 2023-07-10T17:10:06.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: IDENTITY file: IDENTITY 2023-07-10T17:10:06.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 883 Bytes 2023-07-10T17:10:06.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000020.sst 2023-07-10T17:10:06.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000018.log size: 19834116 ; 2023-07-10T17:10:06.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.error_if_exists: 0 2023-07-10T17:10:06.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.create_if_missing: 0 2023-07-10T17:10:06.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.paranoid_checks: 1 2023-07-10T17:10:06.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2023-07-10T17:10:06.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-10T17:10:06.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-10T17:10:06.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.env: 0x5610d6506720 2023-07-10T17:10:06.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.fs: PosixFileSystem 2023-07-10T17:10:06.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.info_log: 0x5610d87af3c0 2023-07-10T17:10:06.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.max_file_opening_threads: 16 2023-07-10T17:10:06.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.statistics: (nil) 2023-07-10T17:10:06.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.use_fsync: 0 2023-07-10T17:10:06.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.max_log_file_size: 0 2023-07-10T17:10:06.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-10T17:10:06.903 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.log_file_time_to_roll: 0 2023-07-10T17:10:06.903 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.keep_log_file_num: 1000 2023-07-10T17:10:06.903 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.recycle_log_file_num: 0 2023-07-10T17:10:06.904 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.allow_fallocate: 1 2023-07-10T17:10:06.904 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.allow_mmap_reads: 0 2023-07-10T17:10:06.904 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.allow_mmap_writes: 0 2023-07-10T17:10:06.904 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.use_direct_reads: 0 2023-07-10T17:10:06.905 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-10T17:10:06.905 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.create_missing_column_families: 0 2023-07-10T17:10:06.905 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.db_log_dir: 2023-07-10T17:10:06.905 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.wal_dir: 2023-07-10T17:10:06.906 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.table_cache_numshardbits: 6 2023-07-10T17:10:06.906 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-07-10T17:10:06.906 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-07-10T17:10:06.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-10T17:10:06.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-10T17:10:06.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-07-10T17:10:06.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.advise_random_on_open: 1 2023-07-10T17:10:06.908 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.db_write_buffer_size: 0 2023-07-10T17:10:06.908 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.write_buffer_manager: 0x5610d7a3cc80 2023-07-10T17:10:06.908 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-10T17:10:06.908 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-10T17:10:06.909 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.use_adaptive_mutex: 0 2023-07-10T17:10:06.909 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.rate_limiter: (nil) 2023-07-10T17:10:06.909 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-10T17:10:06.909 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.wal_recovery_mode: 2 2023-07-10T17:10:06.910 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.enable_thread_tracking: 0 2023-07-10T17:10:06.910 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.enable_pipelined_write: 0 2023-07-10T17:10:06.910 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.unordered_write: 0 2023-07-10T17:10:06.911 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-10T17:10:06.911 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-10T17:10:06.911 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-10T17:10:06.911 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-10T17:10:06.911 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.row_cache: None 2023-07-10T17:10:06.911 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.wal_filter: None 2023-07-10T17:10:06.912 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-10T17:10:06.912 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.allow_ingest_behind: 0 2023-07-10T17:10:06.912 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.two_write_queues: 0 2023-07-10T17:10:06.912 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.manual_wal_flush: 0 2023-07-10T17:10:06.912 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.wal_compression: 0 2023-07-10T17:10:06.912 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.atomic_flush: 0 2023-07-10T17:10:06.912 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-10T17:10:06.913 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.persist_stats_to_disk: 0 2023-07-10T17:10:06.913 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-07-10T17:10:06.913 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.log_readahead_size: 0 2023-07-10T17:10:06.913 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-10T17:10:06.913 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.best_efforts_recovery: 0 2023-07-10T17:10:06.913 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-10T17:10:06.913 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-10T17:10:06.914 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.allow_data_in_errors: 0 2023-07-10T17:10:06.914 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.db_host_id: __hostname__ 2023-07-10T17:10:06.914 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.enforce_single_del_contracts: true 2023-07-10T17:10:06.914 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.max_background_jobs: 2 2023-07-10T17:10:06.914 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.max_background_compactions: -1 2023-07-10T17:10:06.914 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.max_subcompactions: 1 2023-07-10T17:10:06.914 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-10T17:10:06.914 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-10T17:10:06.915 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.delayed_write_rate : 16777216 2023-07-10T17:10:06.915 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.max_total_wal_size: 0 2023-07-10T17:10:06.915 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-10T17:10:06.915 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.stats_dump_period_sec: 600 2023-07-10T17:10:06.915 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.stats_persist_period_sec: 600 2023-07-10T17:10:06.916 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-10T17:10:06.916 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.max_open_files: -1 2023-07-10T17:10:06.916 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.bytes_per_sync: 0 2023-07-10T17:10:06.916 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-07-10T17:10:06.916 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-07-10T17:10:06.917 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.compaction_readahead_size: 0 2023-07-10T17:10:06.917 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.max_background_flushes: -1 2023-07-10T17:10:06.917 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Compression algorithms supported: 2023-07-10T17:10:06.918 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-10T17:10:06.918 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: kZSTD supported: 0 2023-07-10T17:10:06.918 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: kXpressCompression supported: 0 2023-07-10T17:10:06.918 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: kLZ4HCCompression supported: 1 2023-07-10T17:10:06.919 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: kZlibCompression supported: 1 2023-07-10T17:10:06.919 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: kSnappyCompression supported: 1 2023-07-10T17:10:06.919 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: kLZ4Compression supported: 1 2023-07-10T17:10:06.919 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: kBZip2Compression supported: 0 2023-07-10T17:10:06.920 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-07-10T17:10:06.920 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2023-07-10T17:10:06.920 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-07-10T17:10:06.921 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-10T17:10:06.921 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-10T17:10:06.921 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.merge_operator: 2023-07-10T17:10:06.922 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.compaction_filter: None 2023-07-10T17:10:06.922 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.compaction_filter_factory: None 2023-07-10T17:10:06.922 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.sst_partitioner_factory: None 2023-07-10T17:10:06.922 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-07-10T17:10:06.923 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.table_factory: BlockBasedTable 2023-07-10T17:10:06.923 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5610d87af500) 2023-07-10T17:10:06.923 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: cache_index_and_filter_blocks: 1 2023-07-10T17:10:06.923 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-10T17:10:06.924 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-10T17:10:06.924 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: pin_top_level_index_and_filter: 1 2023-07-10T17:10:06.924 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: index_type: 0 2023-07-10T17:10:06.925 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: data_block_index_type: 0 2023-07-10T17:10:06.925 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: index_shortening: 1 2023-07-10T17:10:06.925 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: data_block_hash_table_util_ratio: 0.750000 2023-07-10T17:10:06.925 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: checksum: 4 2023-07-10T17:10:06.926 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: no_block_cache: 0 2023-07-10T17:10:06.926 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: block_cache: 0x5610d7abf1f0 2023-07-10T17:10:06.926 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: block_cache_name: BinnedLRUCache 2023-07-10T17:10:06.926 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: block_cache_options: 2023-07-10T17:10:06.926 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: capacity : 536870912 2023-07-10T17:10:06.926 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: num_shard_bits : 4 2023-07-10T17:10:06.926 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: strict_capacity_limit : 0 2023-07-10T17:10:06.926 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: high_pri_pool_ratio: 0.000 2023-07-10T17:10:06.927 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: block_cache_compressed: (nil) 2023-07-10T17:10:06.927 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: persistent_cache: (nil) 2023-07-10T17:10:06.927 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: block_size: 4096 2023-07-10T17:10:06.927 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: block_size_deviation: 10 2023-07-10T17:10:06.927 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: block_restart_interval: 16 2023-07-10T17:10:06.927 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: index_block_restart_interval: 1 2023-07-10T17:10:06.928 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: metadata_block_size: 4096 2023-07-10T17:10:06.928 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: partition_filters: 0 2023-07-10T17:10:06.928 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: use_delta_encoding: 1 2023-07-10T17:10:06.928 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: filter_policy: bloomfilter 2023-07-10T17:10:06.928 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: whole_key_filtering: 1 2023-07-10T17:10:06.928 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: verify_compression: 0 2023-07-10T17:10:06.929 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: read_amp_bytes_per_bit: 0 2023-07-10T17:10:06.929 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: format_version: 5 2023-07-10T17:10:06.929 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: enable_index_compression: 1 2023-07-10T17:10:06.929 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: block_align: 0 2023-07-10T17:10:06.929 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: max_auto_readahead_size: 262144 2023-07-10T17:10:06.929 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: prepopulate_block_cache: 0 2023-07-10T17:10:06.929 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: initial_auto_readahead_size: 8192 2023-07-10T17:10:06.930 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: num_file_reads_for_auto_readahead: 2 2023-07-10T17:10:06.930 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.write_buffer_size: 33554432 2023-07-10T17:10:06.930 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.max_write_buffer_number: 2 2023-07-10T17:10:06.930 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.compression: NoCompression 2023-07-10T17:10:06.930 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.bottommost_compression: Disabled 2023-07-10T17:10:06.930 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.545+0000 7fd503378c80 4 rocksdb: Options.prefix_extractor: nullptr 2023-07-10T17:10:06.930 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-10T17:10:06.931 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.num_levels: 7 2023-07-10T17:10:06.931 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-10T17:10:06.931 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-10T17:10:06.931 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-10T17:10:06.931 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-10T17:10:06.931 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-10T17:10:06.932 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-10T17:10:06.932 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-10T17:10:06.932 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-10T17:10:06.932 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-10T17:10:06.932 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-10T17:10:06.932 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-10T17:10:06.932 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-10T17:10:06.933 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compression_opts.window_bits: -14 2023-07-10T17:10:06.933 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compression_opts.level: 32767 2023-07-10T17:10:06.933 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compression_opts.strategy: 0 2023-07-10T17:10:06.933 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-10T17:10:06.933 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-10T17:10:06.933 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-10T17:10:06.933 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-10T17:10:06.933 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compression_opts.enabled: false 2023-07-10T17:10:06.934 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-10T17:10:06.934 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-10T17:10:06.934 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-10T17:10:06.934 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-10T17:10:06.934 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.target_file_size_base: 67108864 2023-07-10T17:10:06.934 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.target_file_size_multiplier: 1 2023-07-10T17:10:06.934 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-10T17:10:06.935 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-10T17:10:06.935 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-10T17:10:06.935 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-10T17:10:06.935 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-10T17:10:06.935 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-10T17:10:06.935 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-10T17:10:06.935 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-10T17:10:06.935 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-10T17:10:06.936 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-10T17:10:06.936 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-10T17:10:06.936 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-10T17:10:06.936 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-10T17:10:06.936 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.arena_block_size: 1048576 2023-07-10T17:10:06.936 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-10T17:10:06.936 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-10T17:10:06.936 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.disable_auto_compactions: 0 2023-07-10T17:10:06.937 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-10T17:10:06.937 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-10T17:10:06.937 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-10T17:10:06.937 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-10T17:10:06.937 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-10T17:10:06.937 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-10T17:10:06.937 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-10T17:10:06.938 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-10T17:10:06.938 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-10T17:10:06.938 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-10T17:10:06.938 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.table_properties_collectors: 2023-07-10T17:10:06.938 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.inplace_update_support: 0 2023-07-10T17:10:06.938 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-07-10T17:10:06.938 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-10T17:10:06.938 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-10T17:10:06.939 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.memtable_huge_page_size: 0 2023-07-10T17:10:06.939 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.bloom_locality: 0 2023-07-10T17:10:06.939 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.max_successive_merges: 0 2023-07-10T17:10:06.939 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-07-10T17:10:06.939 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.paranoid_file_checks: 0 2023-07-10T17:10:06.939 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.force_consistency_checks: 1 2023-07-10T17:10:06.939 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.report_bg_io_stats: 0 2023-07-10T17:10:06.939 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.ttl: 2592000 2023-07-10T17:10:06.940 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-07-10T17:10:06.940 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-10T17:10:06.940 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-10T17:10:06.940 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.enable_blob_files: false 2023-07-10T17:10:06.940 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.min_blob_size: 0 2023-07-10T17:10:06.940 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.blob_file_size: 268435456 2023-07-10T17:10:06.940 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.blob_compression_type: NoCompression 2023-07-10T17:10:06.941 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-07-10T17:10:06.941 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-10T17:10:06.941 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-10T17:10:06.941 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-10T17:10:06.941 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.blob_file_starting_level: 0 2023-07-10T17:10:06.941 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-10T17:10:06.941 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd4f8b97700 3 rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 20.sst 2023-07-10T17:10:06.942 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-07-10T17:10:06.942 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 22, last_sequence is 12027, log_number is 18,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-10T17:10:06.942 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 18 2023-07-10T17:10:06.942 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2968d0dc-e328-4528-9a71-860c61a2319b 2023-07-10T17:10:06.942 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1689009006553089, "job": 1, "event": "recovery_started", "wal_files": [18]} 2023-07-10T17:10:06.943 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.549+0000 7fd503378c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #18 mode 2 2023-07-10T17:10:06.943 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.601+0000 7fd503378c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1689009006607089, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 23, "file_size": 12787055, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12033, "largest_seqno": 15256, "table_properties": {"data_size": 12770509, "index_size": 12773, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2757, "raw_key_size": 20847, "raw_average_key_size": 19, "raw_value_size": 12742527, "raw_average_value_size": 11809, "num_data_blocks": 560, "num_entries": 1079, "num_filter_entries": 1079, "num_deletions": 253, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689009006, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2968d0dc-e328-4528-9a71-860c61a2319b", "db_session_id": "2QMPLLEXFH2M63G0EV4L", "orig_file_number": 23, "seqno_to_time_mapping": "N/A"}} 2023-07-10T17:10:06.943 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.605+0000 7fd503378c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1689009006607253, "job": 1, "event": "recovery_finished"} 2023-07-10T17:10:06.943 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.605+0000 7fd503378c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 25 2023-07-10T17:10:06.943 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.605+0000 7fd503378c80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-07-10T17:10:06.943 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.609+0000 7fd503378c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000018.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-10T17:10:06.943 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.609+0000 7fd503378c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5610d7b5e000 2023-07-10T17:10:06.943 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.609+0000 7fd503378c80 4 rocksdb: DB pointer 0x5610d7b4c000 2023-07-10T17:10:06.944 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.613+0000 7fd4f1b89700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-10T17:10:06.944 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.613+0000 7fd4f1b89700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-10T17:10:06.944 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: ** DB Stats ** 2023-07-10T17:10:06.944 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Uptime(secs): 0.1 total, 0.1 interval 2023-07-10T17:10:06.944 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-10T17:10:06.944 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-10T17:10:06.944 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-10T17:10:06.945 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-10T17:10:06.945 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-10T17:10:06.945 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-10T17:10:06.945 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: ** Compaction Stats [default] ** 2023-07-10T17:10:06.945 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-10T17:10:06.945 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-10T17:10:06.945 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: L0 1/0 12.19 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 497.8 0.02 0.00 1 0.024 0 0 0.0 0.0 2023-07-10T17:10:06.945 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: L6 1/0 23.29 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2023-07-10T17:10:06.946 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Sum 2/0 35.49 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 497.8 0.02 0.00 1 0.024 0 0 0.0 0.0 2023-07-10T17:10:06.946 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 497.8 0.02 0.00 1 0.024 0 0 0.0 0.0 2023-07-10T17:10:06.946 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: ** Compaction Stats [default] ** 2023-07-10T17:10:06.946 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-10T17:10:06.946 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-10T17:10:06.946 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 497.8 0.02 0.00 1 0.024 0 0 0.0 0.0 2023-07-10T17:10:06.946 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-10T17:10:06.947 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Uptime(secs): 0.1 total, 0.1 interval 2023-07-10T17:10:06.947 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Flush(GB): cumulative 0.012, interval 0.012 2023-07-10T17:10:06.947 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-10T17:10:06.947 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: AddFile(Total Files): cumulative 0, interval 0 2023-07-10T17:10:06.947 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: AddFile(L0 Files): cumulative 0, interval 0 2023-07-10T17:10:06.947 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: AddFile(Keys): cumulative 0, interval 0 2023-07-10T17:10:06.947 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Cumulative compaction: 0.01 GB write, 189.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-10T17:10:06.947 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Interval compaction: 0.01 GB write, 189.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-10T17:10:06.948 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-10T17:10:06.948 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Block cache BinnedLRUCache@0x5610d7abf1f0#7 capacity: 512.00 MB usage: 15.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 3.2e-05 secs_since: 0 2023-07-10T17:10:06.948 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: Block cache entry stats(count,size,portion): FilterBlock(1,2.78 KB,0.000530481%) IndexBlock(1,13.08 KB,0.00249445%) Misc(1,0.00 KB,0%) 2023-07-10T17:10:06.948 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: ** File Read Latency Histogram By Level [default] ** 2023-07-10T17:10:06.948 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.613+0000 7fd503378c80 0 starting mon.c rank 1 at public addrs [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] at bind addrs [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T17:10:06.948 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.613+0000 7fd503378c80 1 mon.c@-1(???) e3 preinit fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T17:10:06.948 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.613+0000 7fd503378c80 0 mon.c@-1(???).mds e1 new map 2023-07-10T17:10:06.948 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.613+0000 7fd503378c80 0 mon.c@-1(???).mds e1 print_map 2023-07-10T17:10:06.949 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: e1 2023-07-10T17:10:06.949 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: enable_multiple, ever_enabled_multiple: 1,1 2023-07-10T17:10:06.949 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-10T17:10:06.949 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: legacy client fscid: -1 2023-07-10T17:10:06.949 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: 2023-07-10T17:10:06.949 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: No filesystems configured 2023-07-10T17:10:06.949 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.613+0000 7fd503378c80 0 mon.c@-1(???).osd e77 crush map has features 3314933000854323200, adjusting msgr requires 2023-07-10T17:10:06.950 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.613+0000 7fd503378c80 0 mon.c@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2023-07-10T17:10:06.950 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.613+0000 7fd503378c80 0 mon.c@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2023-07-10T17:10:06.950 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.613+0000 7fd503378c80 0 mon.c@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2023-07-10T17:10:06.950 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:06 smithi116 bash[65980]: debug 2023-07-10T17:10:06.617+0000 7fd503378c80 1 mon.c@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2023-07-10T17:10:08.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:08 smithi192 bash[23730]: cluster 2023-07-10T17:10:06.833426+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2023-07-10T17:10:08.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:08 smithi192 bash[23730]: cluster 2023-07-10T17:10:07.109747+0000 mon.a (mon.0) 68 : cluster [INF] mon.a calling monitor election 2023-07-10T17:10:08.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:08 smithi192 bash[23730]: cluster 2023-07-10T17:10:07.113008+0000 mon.a (mon.0) 69 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-10T17:10:08.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:08 smithi192 bash[23730]: cluster 2023-07-10T17:10:07.118390+0000 mgr.y (mgr.25390) 26 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:08.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:08 smithi192 bash[23730]: cluster 2023-07-10T17:10:07.125193+0000 mon.a (mon.0) 70 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0]} removed_ranks: {} 2023-07-10T17:10:08.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:08 smithi192 bash[23730]: cluster 2023-07-10T17:10:07.126355+0000 mon.a (mon.0) 71 : cluster [DBG] fsmap 2023-07-10T17:10:08.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:08 smithi192 bash[23730]: cluster 2023-07-10T17:10:07.126413+0000 mon.a (mon.0) 72 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2023-07-10T17:10:08.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:08 smithi192 bash[23730]: cluster 2023-07-10T17:10:07.127024+0000 mon.a (mon.0) 73 : cluster [DBG] mgrmap e31: y(active, since 25s), standbys: x 2023-07-10T17:10:08.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:08 smithi192 bash[23730]: cluster 2023-07-10T17:10:07.134122+0000 mon.a (mon.0) 74 : cluster [WRN] Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:10:08.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:08 smithi192 bash[23730]: cluster 2023-07-10T17:10:07.134153+0000 mon.a (mon.0) 75 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:10:08.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:08 smithi192 bash[23730]: cluster 2023-07-10T17:10:07.134167+0000 mon.a (mon.0) 76 : cluster [WRN] daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:10:08.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:08 smithi192 bash[23730]: cluster 2023-07-10T17:10:07.134184+0000 mon.a (mon.0) 77 : cluster [WRN] daemon prometheus.a on smithi192 is in error state 2023-07-10T17:10:08.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:08 smithi192 bash[23730]: audit 2023-07-10T17:10:07.139795+0000 mon.a (mon.0) 78 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:08.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:08 smithi192 bash[23730]: audit 2023-07-10T17:10:07.147948+0000 mon.a (mon.0) 79 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:10.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:10 smithi192 bash[23730]: cluster 2023-07-10T17:10:09.120105+0000 mgr.y (mgr.25390) 27 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:11.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:11 smithi192 bash[23730]: audit 2023-07-10T17:10:10.752894+0000 mon.a (mon.0) 80 : audit [DBG] from='client.34233 172.21.15.116:0/571250513' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:11.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:11 smithi192 bash[23730]: audit 2023-07-10T17:10:11.128461+0000 mon.a (mon.0) 81 : audit [INF] from='client.34239 172.21.15.116:0/914235162' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: access denied 2023-07-10T17:10:11.893 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:10:11 smithi116 bash[59394]: debug 2023-07-10T17:10:11.625+0000 7f69d6487700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-07-10T17:10:12.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:12 smithi192 bash[23730]: cluster 2023-07-10T17:10:11.120975+0000 mgr.y (mgr.25390) 28 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:13.137 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:12 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Scheduled restart job, restart counter is at 6. 2023-07-10T17:10:13.137 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:12 smithi192 systemd[1]: Stopped Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:10:13.137 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:12 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Start request repeated too quickly. 2023-07-10T17:10:13.137 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:12 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service: Failed with result 'exit-code'. 2023-07-10T17:10:13.137 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:10:12 smithi192 systemd[1]: Failed to start Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:10:13.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:13 smithi192 bash[23730]: cluster 2023-07-10T17:10:13.122365+0000 mgr.y (mgr.25390) 29 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:15.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:15 smithi192 bash[23730]: audit 2023-07-10T17:10:14.083352+0000 mon.a (mon.0) 82 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:15.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:15 smithi192 bash[23730]: audit 2023-07-10T17:10:14.093126+0000 mon.a (mon.0) 83 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:15.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:15 smithi192 bash[23730]: audit 2023-07-10T17:10:14.890380+0000 mon.a (mon.0) 84 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:15.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:15 smithi192 bash[23730]: audit 2023-07-10T17:10:14.901650+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:16.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:16 smithi192 bash[23730]: cluster 2023-07-10T17:10:15.123070+0000 mgr.y (mgr.25390) 30 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:18.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:18 smithi192 bash[23730]: cluster 2023-07-10T17:10:17.123865+0000 mgr.y (mgr.25390) 31 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:20.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:20 smithi192 bash[23730]: cluster 2023-07-10T17:10:19.125662+0000 mgr.y (mgr.25390) 32 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:22.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:22 smithi192 bash[23730]: cluster 2023-07-10T17:10:21.126628+0000 mgr.y (mgr.25390) 33 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:23.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:23 smithi192 bash[23730]: cluster 2023-07-10T17:10:23.128342+0000 mgr.y (mgr.25390) 34 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:26.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:26 smithi192 bash[23730]: cluster 2023-07-10T17:10:25.129020+0000 mgr.y (mgr.25390) 35 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:26.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:26 smithi192 bash[23730]: audit 2023-07-10T17:10:25.274014+0000 mon.a (mon.0) 86 : audit [DBG] from='client.34257 172.21.15.116:0/218827705' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:26.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:26 smithi192 bash[23730]: audit 2023-07-10T17:10:25.454105+0000 mon.a (mon.0) 87 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:26.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:26 smithi192 bash[23730]: audit 2023-07-10T17:10:25.466964+0000 mon.a (mon.0) 88 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:26.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:26 smithi192 bash[23730]: audit 2023-07-10T17:10:25.468771+0000 mon.a (mon.0) 89 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:26.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:26 smithi192 bash[23730]: audit 2023-07-10T17:10:25.469941+0000 mon.a (mon.0) 90 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:10:26.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:26 smithi192 bash[23730]: audit 2023-07-10T17:10:25.477958+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:26.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:26 smithi192 bash[23730]: audit 2023-07-10T17:10:25.562669+0000 mon.a (mon.0) 92 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:10:26.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:26 smithi192 bash[23730]: audit 2023-07-10T17:10:25.566356+0000 mon.a (mon.0) 93 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:10:26.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:26 smithi192 bash[23730]: audit 2023-07-10T17:10:25.567937+0000 mon.a (mon.0) 94 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-10T17:10:26.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:26 smithi192 bash[23730]: audit 2023-07-10T17:10:25.568937+0000 mon.a (mon.0) 95 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2023-07-10T17:10:26.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:26 smithi192 bash[23730]: audit 2023-07-10T17:10:25.654742+0000 mon.a (mon.0) 96 : audit [INF] from='client.34263 172.21.15.116:0/1098552669' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: access denied 2023-07-10T17:10:27.235 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:27 smithi192 bash[23730]: cephadm 2023-07-10T17:10:25.470917+0000 mgr.y (mgr.25390) 36 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:27.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:27 smithi192 bash[23730]: cluster 2023-07-10T17:10:25.471870+0000 mgr.y (mgr.25390) 37 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:27.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:27 smithi192 bash[23730]: cluster 2023-07-10T17:10:25.472185+0000 mgr.y (mgr.25390) 38 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:27.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:27 smithi192 bash[23730]: cephadm 2023-07-10T17:10:25.569879+0000 mgr.y (mgr.25390) 39 : cephadm [INF] Upgrade: It appears safe to stop mon.b 2023-07-10T17:10:29.235 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:28 smithi192 bash[23730]: cluster 2023-07-10T17:10:27.473786+0000 mgr.y (mgr.25390) 40 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2023-07-10T17:10:29.235 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:28 smithi192 bash[23730]: cephadm 2023-07-10T17:10:27.940293+0000 mgr.y (mgr.25390) 41 : cephadm [INF] Upgrade: Updating mon.b 2023-07-10T17:10:29.235 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:28 smithi192 bash[23730]: audit 2023-07-10T17:10:27.951774+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:29.235 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:28 smithi192 bash[23730]: audit 2023-07-10T17:10:27.953582+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:10:29.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:28 smithi192 bash[23730]: audit 2023-07-10T17:10:27.955194+0000 mon.a (mon.0) 99 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:10:29.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:28 smithi192 bash[23730]: audit 2023-07-10T17:10:27.956667+0000 mon.a (mon.0) 100 : audit [DBG] from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:29.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:28 smithi192 bash[23730]: cephadm 2023-07-10T17:10:27.958349+0000 mgr.y (mgr.25390) 42 : cephadm [INF] Deploying daemon mon.b on smithi192 2023-07-10T17:10:29.408 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:10:30.153 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:10:30.153 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (3m) 16s ago 9m 10.8M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:10:30.154 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (2m) 41s ago 9m 25.4M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:10:30.154 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 error 16s ago 7m - - 2023-07-10T17:10:30.154 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (84s) 41s ago 14m 426M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:10:30.154 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (2m) 16s ago 16m 464M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:10:30.154 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (58s) 16s ago 16m 45.2M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:10:30.154 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (14m) 41s ago 14m 69.1M 2048M 16.2.0 24ecd6d5f14c 9ad4329e5488 2023-07-10T17:10:30.154 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (24s) 16s ago 14m 38.1M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:10:30.154 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (3m) 16s ago 10m 9287k - 0.18.1 e5a616e4b9cf 3bd058d91353 2023-07-10T17:10:30.155 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (3m) 41s ago 10m 3563k - 0.18.1 e5a616e4b9cf 63551615c679 2023-07-10T17:10:30.155 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (13m) 16s ago 13m 55.4M 4096M 16.2.0 24ecd6d5f14c 4fa26367c830 2023-07-10T17:10:30.155 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (13m) 16s ago 13m 53.6M 4096M 16.2.0 24ecd6d5f14c c157509b3fd4 2023-07-10T17:10:30.155 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (12m) 16s ago 13m 49.4M 4096M 16.2.0 24ecd6d5f14c f1fd77b3c454 2023-07-10T17:10:30.155 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (12m) 16s ago 12m 54.9M 4096M 16.2.0 24ecd6d5f14c 60949d479c11 2023-07-10T17:10:30.155 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (12m) 41s ago 12m 54.5M 4096M 16.2.0 24ecd6d5f14c 75fc0b36aab8 2023-07-10T17:10:30.155 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (11m) 41s ago 11m 52.4M 4096M 16.2.0 24ecd6d5f14c d673065333dd 2023-07-10T17:10:30.156 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (11m) 41s ago 11m 53.3M 4096M 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:10:30.156 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (10m) 41s ago 11m 56.7M 4096M 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:10:30.156 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 error 41s ago 10m - - 2023-07-10T17:10:30.156 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (7m) 16s ago 7m 54.7M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:10:30.156 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (7m) 41s ago 7m 54.3M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:10:30.156 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (7m) 16s ago 7m 54.6M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:10:30.156 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (7m) 41s ago 7m 56.2M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:10:30.329 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:30 smithi192 systemd[1]: Stopping Ceph mon.b for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:10:30.595 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:10:30.595 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:10:30.595 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2023-07-10T17:10:30.595 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:10:30.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:10:30.596 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:10:30.596 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:10:30.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:10:30.596 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:10:30.596 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-10T17:10:30.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:10:30.596 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:10:30.597 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:10:30.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:10:30.597 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:10:30.597 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 13, 2023-07-10T17:10:30.597 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 4 2023-07-10T17:10:30.597 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:10:30.597 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:10:30.636 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:30 smithi192 bash[23730]: debug 2023-07-10T17:10:30.321+0000 7f172ea87700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-07-10T17:10:30.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:30 smithi192 bash[23730]: debug 2023-07-10T17:10:30.321+0000 7f172ea87700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-07-10T17:10:30.965 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:10:30.966 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:10:30.966 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:10:30.966 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:10:30.966 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:10:30.966 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2023-07-10T17:10:30.967 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:10:30.967 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "4/23 daemons upgraded", 2023-07-10T17:10:30.967 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading mon daemons", 2023-07-10T17:10:30.967 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:10:30.967 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:10:31.363 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:31 smithi192 bash[50202]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-mon.b 2023-07-10T17:10:31.363 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:31 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.b.service: Succeeded. 2023-07-10T17:10:31.363 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:31 smithi192 systemd[1]: Stopped Ceph mon.b for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:10:31.388 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:10:31.388 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:10:31.388 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:10:31.388 INFO:teuthology.orchestra.run.smithi116.stdout: daemon prometheus.a on smithi192 is in error state 2023-07-10T17:10:31.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:31 smithi192 systemd[1]: Started Ceph mon.b for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:10:32.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.461+0000 7f9ed58ffc80 0 set uid:gid to 167:167 (ceph:ceph) 2023-07-10T17:10:32.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.461+0000 7f9ed58ffc80 0 ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc), process ceph-mon, pid 7 2023-07-10T17:10:32.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.461+0000 7f9ed58ffc80 0 pidfile_write: ignore empty --pid-file 2023-07-10T17:10:32.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 0 load: jerasure load: lrc 2023-07-10T17:10:32.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: RocksDB version: 7.9.2 2023-07-10T17:10:32.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Git sha 0 2023-07-10T17:10:32.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Compile date 2023-07-07 23:11:19 2023-07-10T17:10:32.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: DB SUMMARY 2023-07-10T17:10:32.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: DB Session ID: SW4IFCIIIJMPULFU2TY4 2023-07-10T17:10:32.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: CURRENT file: CURRENT 2023-07-10T17:10:32.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: IDENTITY file: IDENTITY 2023-07-10T17:10:32.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 883 Bytes 2023-07-10T17:10:32.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000020.sst 2023-07-10T17:10:32.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000018.log size: 21985960 ; 2023-07-10T17:10:32.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.error_if_exists: 0 2023-07-10T17:10:32.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.create_if_missing: 0 2023-07-10T17:10:32.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.paranoid_checks: 1 2023-07-10T17:10:32.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.flush_verify_memtable_count: 1 2023-07-10T17:10:32.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-10T17:10:32.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-10T17:10:32.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.env: 0x555928377720 2023-07-10T17:10:32.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.fs: PosixFileSystem 2023-07-10T17:10:32.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.info_log: 0x555929b093c0 2023-07-10T17:10:32.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.max_file_opening_threads: 16 2023-07-10T17:10:32.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.statistics: (nil) 2023-07-10T17:10:32.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.use_fsync: 0 2023-07-10T17:10:32.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.max_log_file_size: 0 2023-07-10T17:10:32.642 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-10T17:10:32.643 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.log_file_time_to_roll: 0 2023-07-10T17:10:32.643 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.keep_log_file_num: 1000 2023-07-10T17:10:32.643 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.recycle_log_file_num: 0 2023-07-10T17:10:32.643 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.allow_fallocate: 1 2023-07-10T17:10:32.643 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.allow_mmap_reads: 0 2023-07-10T17:10:32.643 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.allow_mmap_writes: 0 2023-07-10T17:10:32.644 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.use_direct_reads: 0 2023-07-10T17:10:32.644 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-10T17:10:32.644 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.create_missing_column_families: 0 2023-07-10T17:10:32.644 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.db_log_dir: 2023-07-10T17:10:32.644 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.wal_dir: 2023-07-10T17:10:32.644 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.table_cache_numshardbits: 6 2023-07-10T17:10:32.644 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-07-10T17:10:32.645 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-07-10T17:10:32.645 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-10T17:10:32.645 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-10T17:10:32.645 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-07-10T17:10:32.645 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.advise_random_on_open: 1 2023-07-10T17:10:32.645 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.db_write_buffer_size: 0 2023-07-10T17:10:32.645 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.write_buffer_manager: 0x555928d96c80 2023-07-10T17:10:32.646 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-10T17:10:32.646 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-10T17:10:32.646 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.use_adaptive_mutex: 0 2023-07-10T17:10:32.646 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.rate_limiter: (nil) 2023-07-10T17:10:32.646 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-10T17:10:32.646 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.wal_recovery_mode: 2 2023-07-10T17:10:32.646 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.enable_thread_tracking: 0 2023-07-10T17:10:32.647 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.enable_pipelined_write: 0 2023-07-10T17:10:32.647 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.unordered_write: 0 2023-07-10T17:10:32.647 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-10T17:10:32.647 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-10T17:10:32.647 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-10T17:10:32.648 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-10T17:10:32.648 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.row_cache: None 2023-07-10T17:10:32.648 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.wal_filter: None 2023-07-10T17:10:32.648 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-10T17:10:32.648 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.allow_ingest_behind: 0 2023-07-10T17:10:32.648 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.two_write_queues: 0 2023-07-10T17:10:32.649 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.manual_wal_flush: 0 2023-07-10T17:10:32.649 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.wal_compression: 0 2023-07-10T17:10:32.649 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.atomic_flush: 0 2023-07-10T17:10:32.649 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-10T17:10:32.649 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.persist_stats_to_disk: 0 2023-07-10T17:10:32.649 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-07-10T17:10:32.649 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.log_readahead_size: 0 2023-07-10T17:10:32.649 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-10T17:10:32.650 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.best_efforts_recovery: 0 2023-07-10T17:10:32.650 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-10T17:10:32.650 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-10T17:10:32.650 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.allow_data_in_errors: 0 2023-07-10T17:10:32.650 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.db_host_id: __hostname__ 2023-07-10T17:10:32.650 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.enforce_single_del_contracts: true 2023-07-10T17:10:32.650 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.max_background_jobs: 2 2023-07-10T17:10:32.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.max_background_compactions: -1 2023-07-10T17:10:32.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.max_subcompactions: 1 2023-07-10T17:10:32.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-10T17:10:32.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-10T17:10:32.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.delayed_write_rate : 16777216 2023-07-10T17:10:32.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.max_total_wal_size: 0 2023-07-10T17:10:32.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-10T17:10:32.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.stats_dump_period_sec: 600 2023-07-10T17:10:32.652 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.stats_persist_period_sec: 600 2023-07-10T17:10:32.652 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-10T17:10:32.652 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.max_open_files: -1 2023-07-10T17:10:32.652 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.bytes_per_sync: 0 2023-07-10T17:10:32.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-07-10T17:10:32.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-07-10T17:10:32.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.compaction_readahead_size: 0 2023-07-10T17:10:32.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Options.max_background_flushes: -1 2023-07-10T17:10:32.654 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Compression algorithms supported: 2023-07-10T17:10:32.654 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-10T17:10:32.654 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: kZSTD supported: 0 2023-07-10T17:10:32.654 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: kXpressCompression supported: 0 2023-07-10T17:10:32.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: kLZ4HCCompression supported: 1 2023-07-10T17:10:32.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: kZlibCompression supported: 1 2023-07-10T17:10:32.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: kSnappyCompression supported: 1 2023-07-10T17:10:32.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: kLZ4Compression supported: 1 2023-07-10T17:10:32.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: kBZip2Compression supported: 0 2023-07-10T17:10:32.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-07-10T17:10:32.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.465+0000 7f9ed58ffc80 4 rocksdb: DMutex implementation: pthread_mutex_t 2023-07-10T17:10:32.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-10T17:10:32.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-10T17:10:32.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-10T17:10:32.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.merge_operator: 2023-07-10T17:10:32.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compaction_filter: None 2023-07-10T17:10:32.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compaction_filter_factory: None 2023-07-10T17:10:32.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.sst_partitioner_factory: None 2023-07-10T17:10:32.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-07-10T17:10:32.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.table_factory: BlockBasedTable 2023-07-10T17:10:32.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555929b09500) 2023-07-10T17:10:32.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: cache_index_and_filter_blocks: 1 2023-07-10T17:10:32.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-10T17:10:32.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-10T17:10:32.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: pin_top_level_index_and_filter: 1 2023-07-10T17:10:32.661 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: index_type: 0 2023-07-10T17:10:32.661 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: data_block_index_type: 0 2023-07-10T17:10:32.661 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: index_shortening: 1 2023-07-10T17:10:32.661 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: data_block_hash_table_util_ratio: 0.750000 2023-07-10T17:10:32.662 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: checksum: 4 2023-07-10T17:10:32.662 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: no_block_cache: 0 2023-07-10T17:10:32.662 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: block_cache: 0x555928e191f0 2023-07-10T17:10:32.662 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: block_cache_name: BinnedLRUCache 2023-07-10T17:10:32.663 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: block_cache_options: 2023-07-10T17:10:32.663 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: capacity : 536870912 2023-07-10T17:10:32.663 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: num_shard_bits : 4 2023-07-10T17:10:32.663 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: strict_capacity_limit : 0 2023-07-10T17:10:32.663 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: high_pri_pool_ratio: 0.000 2023-07-10T17:10:32.663 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: block_cache_compressed: (nil) 2023-07-10T17:10:32.663 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: persistent_cache: (nil) 2023-07-10T17:10:32.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: block_size: 4096 2023-07-10T17:10:32.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: block_size_deviation: 10 2023-07-10T17:10:32.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: block_restart_interval: 16 2023-07-10T17:10:32.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: index_block_restart_interval: 1 2023-07-10T17:10:32.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: metadata_block_size: 4096 2023-07-10T17:10:32.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: partition_filters: 0 2023-07-10T17:10:32.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: use_delta_encoding: 1 2023-07-10T17:10:32.665 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: filter_policy: bloomfilter 2023-07-10T17:10:32.665 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: whole_key_filtering: 1 2023-07-10T17:10:32.665 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: verify_compression: 0 2023-07-10T17:10:32.665 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: read_amp_bytes_per_bit: 0 2023-07-10T17:10:32.665 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: format_version: 5 2023-07-10T17:10:32.665 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: enable_index_compression: 1 2023-07-10T17:10:32.665 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: block_align: 0 2023-07-10T17:10:32.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: max_auto_readahead_size: 262144 2023-07-10T17:10:32.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: prepopulate_block_cache: 0 2023-07-10T17:10:32.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: initial_auto_readahead_size: 8192 2023-07-10T17:10:32.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: num_file_reads_for_auto_readahead: 2 2023-07-10T17:10:32.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.write_buffer_size: 33554432 2023-07-10T17:10:32.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.max_write_buffer_number: 2 2023-07-10T17:10:32.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compression: NoCompression 2023-07-10T17:10:32.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.bottommost_compression: Disabled 2023-07-10T17:10:32.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.prefix_extractor: nullptr 2023-07-10T17:10:32.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-10T17:10:32.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.num_levels: 7 2023-07-10T17:10:32.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-10T17:10:32.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-10T17:10:32.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-10T17:10:32.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-10T17:10:32.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-10T17:10:32.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-10T17:10:32.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-10T17:10:32.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-10T17:10:32.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-10T17:10:32.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-10T17:10:32.669 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-10T17:10:32.669 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-10T17:10:32.669 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compression_opts.window_bits: -14 2023-07-10T17:10:32.669 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compression_opts.level: 32767 2023-07-10T17:10:32.669 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compression_opts.strategy: 0 2023-07-10T17:10:32.669 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-10T17:10:32.669 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-10T17:10:32.669 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-10T17:10:32.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-10T17:10:32.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compression_opts.enabled: false 2023-07-10T17:10:32.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-10T17:10:32.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-10T17:10:32.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-10T17:10:32.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-10T17:10:32.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.target_file_size_base: 67108864 2023-07-10T17:10:32.671 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.target_file_size_multiplier: 1 2023-07-10T17:10:32.671 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-10T17:10:32.671 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-10T17:10:32.671 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-10T17:10:32.671 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-10T17:10:32.671 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-10T17:10:32.671 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-10T17:10:32.671 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-10T17:10:32.672 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-10T17:10:32.672 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-10T17:10:32.672 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-10T17:10:32.672 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-10T17:10:32.672 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-10T17:10:32.672 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-10T17:10:32.672 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.arena_block_size: 1048576 2023-07-10T17:10:32.673 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-10T17:10:32.673 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-10T17:10:32.673 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.disable_auto_compactions: 0 2023-07-10T17:10:32.673 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-10T17:10:32.673 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-10T17:10:32.673 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-10T17:10:32.673 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-10T17:10:32.674 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-10T17:10:32.674 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-10T17:10:32.674 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-10T17:10:32.674 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-10T17:10:32.674 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-10T17:10:32.674 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-10T17:10:32.674 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.table_properties_collectors: 2023-07-10T17:10:32.675 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.inplace_update_support: 0 2023-07-10T17:10:32.675 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-07-10T17:10:32.675 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-10T17:10:32.675 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-10T17:10:32.675 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.memtable_huge_page_size: 0 2023-07-10T17:10:32.675 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.bloom_locality: 0 2023-07-10T17:10:32.675 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.max_successive_merges: 0 2023-07-10T17:10:32.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-07-10T17:10:32.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.paranoid_file_checks: 0 2023-07-10T17:10:32.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.force_consistency_checks: 1 2023-07-10T17:10:32.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.report_bg_io_stats: 0 2023-07-10T17:10:32.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.ttl: 2592000 2023-07-10T17:10:32.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-07-10T17:10:32.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-10T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-10T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.enable_blob_files: false 2023-07-10T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.min_blob_size: 0 2023-07-10T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.blob_file_size: 268435456 2023-07-10T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.blob_compression_type: NoCompression 2023-07-10T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-07-10T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-10T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-10T17:10:32.678 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-10T17:10:32.678 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.blob_file_starting_level: 0 2023-07-10T17:10:32.678 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-10T17:10:32.678 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ecb11e700 3 rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 20.sst 2023-07-10T17:10:32.678 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-07-10T17:10:32.678 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 22, last_sequence is 11965, log_number is 18,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-10T17:10:32.678 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 18 2023-07-10T17:10:32.679 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e4e48cc2-b70d-4918-91b8-518451f0360d 2023-07-10T17:10:32.679 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1689009032475927, "job": 1, "event": "recovery_started", "wal_files": [18]} 2023-07-10T17:10:32.679 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.469+0000 7f9ed58ffc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #18 mode 2 2023-07-10T17:10:32.679 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.529+0000 7f9ed58ffc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1689009032535831, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 23, "file_size": 14150854, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11971, "largest_seqno": 15608, "table_properties": {"data_size": 14132457, "index_size": 14368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3013, "raw_key_size": 22784, "raw_average_key_size": 19, "raw_value_size": 14101482, "raw_average_value_size": 11930, "num_data_blocks": 630, "num_entries": 1182, "num_filter_entries": 1182, "num_deletions": 253, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689009032, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e4e48cc2-b70d-4918-91b8-518451f0360d", "db_session_id": "SW4IFCIIIJMPULFU2TY4", "orig_file_number": 23, "seqno_to_time_mapping": "N/A"}} 2023-07-10T17:10:32.679 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.529+0000 7f9ed58ffc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1689009032536078, "job": 1, "event": "recovery_finished"} 2023-07-10T17:10:32.679 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.529+0000 7f9ed58ffc80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 25 2023-07-10T17:10:32.679 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.529+0000 7f9ed58ffc80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-07-10T17:10:32.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.537+0000 7f9ed58ffc80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000018.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-10T17:10:32.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.537+0000 7f9ed58ffc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555928eb8000 2023-07-10T17:10:32.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.537+0000 7f9ed58ffc80 4 rocksdb: DB pointer 0x555928ea6000 2023-07-10T17:10:32.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.537+0000 7f9ec4110700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-10T17:10:32.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.537+0000 7f9ec4110700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-10T17:10:32.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: ** DB Stats ** 2023-07-10T17:10:32.681 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Uptime(secs): 0.1 total, 0.1 interval 2023-07-10T17:10:32.681 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-10T17:10:32.681 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-10T17:10:32.681 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-10T17:10:32.681 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-10T17:10:32.681 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-10T17:10:32.681 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-10T17:10:32.681 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: ** Compaction Stats [default] ** 2023-07-10T17:10:32.682 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-10T17:10:32.682 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-10T17:10:32.682 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: L0 1/0 13.50 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 598.5 0.02 0.00 1 0.023 0 0 0.0 0.0 2023-07-10T17:10:32.682 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: L6 1/0 23.29 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2023-07-10T17:10:32.682 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Sum 2/0 36.79 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 598.5 0.02 0.00 1 0.023 0 0 0.0 0.0 2023-07-10T17:10:32.682 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 598.5 0.02 0.00 1 0.023 0 0 0.0 0.0 2023-07-10T17:10:32.682 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: ** Compaction Stats [default] ** 2023-07-10T17:10:32.683 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-10T17:10:32.683 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-10T17:10:32.683 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 598.5 0.02 0.00 1 0.023 0 0 0.0 0.0 2023-07-10T17:10:32.683 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-10T17:10:32.683 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Uptime(secs): 0.1 total, 0.1 interval 2023-07-10T17:10:32.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Flush(GB): cumulative 0.013, interval 0.013 2023-07-10T17:10:32.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-10T17:10:32.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: AddFile(Total Files): cumulative 0, interval 0 2023-07-10T17:10:32.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: AddFile(L0 Files): cumulative 0, interval 0 2023-07-10T17:10:32.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: AddFile(Keys): cumulative 0, interval 0 2023-07-10T17:10:32.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Cumulative compaction: 0.01 GB write, 195.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-10T17:10:32.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Interval compaction: 0.01 GB write, 195.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-10T17:10:32.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-10T17:10:32.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Block cache BinnedLRUCache@0x555928e191f0#7 capacity: 512.00 MB usage: 19.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 3.3e-05 secs_since: 0 2023-07-10T17:10:32.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: Block cache entry stats(count,size,portion): FilterBlock(1,3.03 KB,0.000578165%) IndexBlock(1,16.08 KB,0.00306666%) Misc(1,0.00 KB,0%) 2023-07-10T17:10:32.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: ** File Read Latency Histogram By Level [default] ** 2023-07-10T17:10:32.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.537+0000 7f9ed58ffc80 0 starting mon.b rank 2 at public addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] at bind addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T17:10:32.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.541+0000 7f9ed58ffc80 1 mon.b@-1(???) e3 preinit fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T17:10:32.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.541+0000 7f9ed58ffc80 0 mon.b@-1(???).mds e1 new map 2023-07-10T17:10:32.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.541+0000 7f9ed58ffc80 0 mon.b@-1(???).mds e1 print_map 2023-07-10T17:10:32.686 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: e1 2023-07-10T17:10:32.686 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: enable_multiple, ever_enabled_multiple: 1,1 2023-07-10T17:10:32.686 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-10T17:10:32.686 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: legacy client fscid: -1 2023-07-10T17:10:32.686 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: 2023-07-10T17:10:32.686 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: No filesystems configured 2023-07-10T17:10:32.686 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.541+0000 7f9ed58ffc80 0 mon.b@-1(???).osd e77 crush map has features 3314933000854323200, adjusting msgr requires 2023-07-10T17:10:32.687 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.541+0000 7f9ed58ffc80 0 mon.b@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2023-07-10T17:10:32.687 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.541+0000 7f9ed58ffc80 0 mon.b@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2023-07-10T17:10:32.687 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.541+0000 7f9ed58ffc80 0 mon.b@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2023-07-10T17:10:32.687 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:32 smithi192 bash[50362]: debug 2023-07-10T17:10:32.541+0000 7f9ed58ffc80 1 mon.b@-1(???).paxosservice(auth 1..26) refresh upgraded, format 0 -> 3 2023-07-10T17:10:37.893 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:10:37 smithi116 bash[59394]: debug 2023-07-10T17:10:37.553+0000 7f69d6487700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-07-10T17:10:40.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T16:54:21.709221+0000 mon.a (mon.0) 0 : 1 mkfs 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T17:10:40.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:06:42.160726+0000 mgr.x (mgr.24857) 25 : cephadm 1 Reconfiguring node-exporter.a (dependencies changed)... 2023-07-10T17:10:40.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:06:42.161197+0000 mgr.x (mgr.24857) 26 : cephadm 1 Reconfiguring daemon node-exporter.a on smithi116 2023-07-10T17:10:40.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:06:43.279509+0000 mgr.x (mgr.24857) 29 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2023-07-10T17:10:40.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:06:43.317778+0000 mgr.x (mgr.24857) 31 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi116 2023-07-10T17:10:40.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:06:52.953889+0000 mgr.x (mgr.24857) 38 : cephadm 1 Reconfiguring iscsi.foo.smithi116.fkthyl (dependencies changed)... 2023-07-10T17:10:40.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:06:52.969207+0000 mgr.x (mgr.24857) 39 : cephadm 1 Reconfiguring daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:10:40.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:06:55.793158+0000 mgr.x (mgr.24857) 41 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2023-07-10T17:10:40.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:06:56.037644+0000 mgr.x (mgr.24857) 43 : cephadm 1 Reconfiguring daemon prometheus.a on smithi192 2023-07-10T17:10:40.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:07:10.084390+0000 mgr.x (mgr.24857) 51 : cephadm 1 Reconfiguring node-exporter.b (dependencies changed)... 2023-07-10T17:10:40.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:07:10.085244+0000 mgr.x (mgr.24857) 52 : cephadm 1 Reconfiguring daemon node-exporter.b on smithi192 2023-07-10T17:10:40.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:07:11.754186+0000 mgr.x (mgr.24857) 53 : cephadm 1 Reconfiguring grafana.a (dependencies changed)... 2023-07-10T17:10:40.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:07:11.769017+0000 mgr.x (mgr.24857) 54 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2023-07-10T17:10:40.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:07:11.878848+0000 mgr.x (mgr.24857) 56 : cephadm 1 Reconfiguring daemon grafana.a on smithi192 2023-07-10T17:10:40.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:09.031472+0000 mgr.x (mgr.24857) 102 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:10:40.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:09.032983+0000 mgr.x (mgr.24857) 103 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:10:40.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:11.403617+0000 mgr.x (mgr.24857) 105 : cephadm 1 Upgrade: Updating mgr.y 2023-07-10T17:10:40.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:11.423499+0000 mgr.x (mgr.24857) 106 : cephadm 1 Deploying daemon mgr.y on smithi116 2023-07-10T17:10:40.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:32.705714+0000 mgr.x (mgr.24857) 122 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:40.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:32.810042+0000 mgr.x (mgr.24857) 125 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:10:40.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:32.810666+0000 mgr.x (mgr.24857) 126 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:10:40.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:32.813767+0000 mgr.x (mgr.24857) 127 : cephadm 1 Failing over to other MGR 2023-07-10T17:10:40.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:33.786136+0000 mgr.y (mgr.15252) 1 : cephadm 1 inventory: adjusted host smithi192 addr 'smithi192' -> '172.21.15.192' 2023-07-10T17:10:40.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:34.964152+0000 mgr.y (mgr.15252) 3 : cephadm 1 [10/Jul/2023:17:08:34] ENGINE Bus STARTING 2023-07-10T17:10:40.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:35.066231+0000 mgr.y (mgr.15252) 4 : cephadm 1 [10/Jul/2023:17:08:35] ENGINE Serving on http://172.21.15.116:8765 2023-07-10T17:10:40.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:35.193673+0000 mgr.y (mgr.15252) 5 : cephadm 1 [10/Jul/2023:17:08:35] ENGINE Serving on https://172.21.15.116:7150 2023-07-10T17:10:40.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:35.194073+0000 mgr.y (mgr.15252) 6 : cephadm 1 [10/Jul/2023:17:08:35] ENGINE Bus STARTED 2023-07-10T17:10:40.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:51.857316+0000 mgr.y (mgr.15252) 16 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:40.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:51.943607+0000 mgr.y (mgr.15252) 18 : cephadm 1 Reconfiguring iscsi.foo.smithi116.fkthyl (dependencies changed)... 2023-07-10T17:10:40.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:51.966478+0000 mgr.y (mgr.15252) 19 : cephadm 1 Reconfiguring daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:10:40.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:54.286766+0000 mgr.y (mgr.15252) 25 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2023-07-10T17:10:40.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:08:54.529934+0000 mgr.y (mgr.15252) 26 : cephadm 1 Reconfiguring daemon prometheus.a on smithi192 2023-07-10T17:10:40.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:09:01.831647+0000 mgr.y (mgr.15252) 33 : cephadm 1 Upgrade: Updating mgr.x 2023-07-10T17:10:40.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:09:01.841176+0000 mgr.y (mgr.15252) 34 : cephadm 1 Deploying daemon mgr.x on smithi192 2023-07-10T17:10:40.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:09:22.611122+0000 mgr.y (mgr.15252) 46 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:40.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:09:22.715387+0000 mgr.y (mgr.15252) 49 : cephadm 1 Upgrade: Setting container_image for all mgr 2023-07-10T17:10:40.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:09:22.751620+0000 mgr.y (mgr.15252) 50 : cephadm 1 Upgrade: It appears safe to stop mon.a 2023-07-10T17:10:40.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:09:26.328487+0000 mgr.y (mgr.15252) 56 : cephadm 1 Upgrade: Updating mon.a 2023-07-10T17:10:40.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:09:26.343243+0000 mgr.y (mgr.15252) 57 : cephadm 1 Deploying daemon mon.a on smithi116 2023-07-10T17:10:40.394 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:41.215841+0000 osd.0 (osd.0) 3 : cluster 3 failed to encode map e77 with expected crc 2023-07-10T17:10:40.394 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:41.217666+0000 osd.5 (osd.5) 3 : cluster 3 failed to encode map e77 with expected crc 2023-07-10T17:10:40.394 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:41.235415+0000 mon.a (mon.0) 34 : cluster 0 Standby manager daemon x started 2023-07-10T17:10:40.394 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:41.240959+0000 mon.a (mon.0) 35 : cluster 1 Manager daemon y is now available 2023-07-10T17:10:40.394 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:41.298769+0000 osd.7 (osd.7) 3 : cluster 3 failed to encode map e77 with expected crc 2023-07-10T17:10:40.394 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:41.334381+0000 osd.3 (osd.3) 3 : cluster 3 failed to encode map e77 with expected crc 2023-07-10T17:10:40.394 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:41.334961+0000 osd.4 (osd.4) 3 : cluster 3 failed to encode map e77 with expected crc 2023-07-10T17:10:40.394 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:09:41.744716+0000 mgr.y (mgr.25390) 1 : cephadm 1 [10/Jul/2023:17:09:41] ENGINE Bus STARTING 2023-07-10T17:10:40.395 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:09:41.847036+0000 mgr.y (mgr.25390) 2 : cephadm 1 [10/Jul/2023:17:09:41] ENGINE Serving on http://172.21.15.116:8765 2023-07-10T17:10:40.395 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:09:41.974125+0000 mgr.y (mgr.25390) 3 : cephadm 1 [10/Jul/2023:17:09:41] ENGINE Serving on https://172.21.15.116:7150 2023-07-10T17:10:40.395 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:09:41.974317+0000 mgr.y (mgr.25390) 4 : cephadm 1 [10/Jul/2023:17:09:41] ENGINE Bus STARTED 2023-07-10T17:10:40.395 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:42.217941+0000 mon.a (mon.0) 40 : cluster 0 mgrmap e30: y(active, since 1.01682s), standbys: x 2023-07-10T17:10:40.395 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:42.231725+0000 mgr.y (mgr.25390) 5 : cluster 0 pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.395 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:43.210357+0000 mon.a (mon.0) 41 : cluster 1 Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2023-07-10T17:10:40.395 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:43.210394+0000 mon.a (mon.0) 42 : cluster 1 Cluster is now healthy 2023-07-10T17:10:40.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:43.231670+0000 mgr.y (mgr.25390) 6 : cluster 0 pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:43.492770+0000 mon.a (mon.0) 43 : cluster 0 mgrmap e31: y(active, since 2s), standbys: x 2023-07-10T17:10:40.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:45.232364+0000 mgr.y (mgr.25390) 7 : cluster 0 pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:47.233995+0000 mgr.y (mgr.25390) 8 : cluster 0 pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:49.192422+0000 mon.a (mon.0) 50 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:49.198955+0000 mon.a (mon.0) 51 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:49.234683+0000 mgr.y (mgr.25390) 9 : cluster 0 pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:51.235421+0000 mgr.y (mgr.25390) 10 : cluster 0 pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.397 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:53.237029+0000 mgr.y (mgr.25390) 11 : cluster 0 pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.397 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:55.237771+0000 mgr.y (mgr.25390) 12 : cluster 0 pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.397 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:56.426782+0000 mon.c (mon.1) 97 : audit 0 from='client.? 172.21.15.116:0/2649887320' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:40.397 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:56.797829+0000 mon.c (mon.1) 98 : audit 1 from='client.? 172.21.15.116:0/1164294302' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]: access denied 2023-07-10T17:10:40.397 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:56.981724+0000 mgr.y (mgr.25390) 13 : audit 0 from='client.25465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.397 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:57.239327+0000 mgr.y (mgr.25390) 14 : cluster 0 pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.397 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:57.363273+0000 mgr.y (mgr.25390) 15 : audit 0 from='client.25471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.398 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:57.744423+0000 mgr.y (mgr.25390) 16 : audit 0 from='client.25477 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.398 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:58.201338+0000 mon.a (mon.0) 52 : audit 0 from='client.34203 172.21.15.116:0/1768468329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:10:40.398 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:58.573745+0000 mgr.y (mgr.25390) 17 : audit 0 from='client.25489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.398 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:59.008117+0000 mon.c (mon.1) 99 : audit 0 from='client.? 172.21.15.116:0/3069563215' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:10:40.398 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:59.098231+0000 mon.a (mon.0) 53 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.398 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:59.108591+0000 mon.a (mon.0) 54 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.398 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:59.111286+0000 mon.a (mon.0) 55 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:40.398 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:59.112529+0000 mon.a (mon.0) 56 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:10:40.399 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:09:59.113529+0000 mgr.y (mgr.25390) 18 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:40.399 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:09:59.114638+0000 mgr.y (mgr.25390) 19 : cluster 0 pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:40.399 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:59.121473+0000 mon.a (mon.0) 57 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.399 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:59.208871+0000 mon.a (mon.0) 58 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:10:40.399 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:59.214394+0000 mon.a (mon.0) 59 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:10:40.399 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:59.216783+0000 mon.a (mon.0) 60 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-10T17:10:40.399 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:09:59.218328+0000 mon.a (mon.0) 61 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2023-07-10T17:10:40.400 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:09:59.219379+0000 mgr.y (mgr.25390) 20 : cephadm 1 Upgrade: It appears safe to stop mon.c 2023-07-10T17:10:40.400 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:00.000194+0000 mon.a (mon.0) 62 : cluster 1 overall HEALTH_OK 2023-07-10T17:10:40.400 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:00.105153+0000 mon.a (mon.0) 63 : cluster 3 Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:10:40.400 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:01.115429+0000 mgr.y (mgr.25390) 21 : cluster 0 pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:40.400 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:10:01.438560+0000 mgr.y (mgr.25390) 22 : cephadm 1 Upgrade: Updating mon.c 2023-07-10T17:10:40.400 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:01.448292+0000 mon.a (mon.0) 64 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.400 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:01.450164+0000 mon.a (mon.0) 65 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:10:40.400 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:01.452396+0000 mon.a (mon.0) 66 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:10:40.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:01.454067+0000 mon.a (mon.0) 67 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:40.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:10:01.455702+0000 mgr.y (mgr.25390) 23 : cephadm 1 Deploying daemon mon.c on smithi116 2023-07-10T17:10:40.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:03.117083+0000 mgr.y (mgr.25390) 24 : cluster 0 pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:40.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:06.833426+0000 mon.c (mon.1) 1 : cluster 1 mon.c calling monitor election 2023-07-10T17:10:40.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:07.109747+0000 mon.a (mon.0) 68 : cluster 1 mon.a calling monitor election 2023-07-10T17:10:40.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:07.113008+0000 mon.a (mon.0) 69 : cluster 1 mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-10T17:10:40.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:07.118390+0000 mgr.y (mgr.25390) 26 : cluster 0 pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:40.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:07.125193+0000 mon.a (mon.0) 70 : cluster 0 monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0]} removed_ranks: {} 2023-07-10T17:10:40.402 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:07.126355+0000 mon.a (mon.0) 71 : cluster 0 fsmap 2023-07-10T17:10:40.402 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:07.126413+0000 mon.a (mon.0) 72 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2023-07-10T17:10:40.402 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:07.127024+0000 mon.a (mon.0) 73 : cluster 0 mgrmap e31: y(active, since 25s), standbys: x 2023-07-10T17:10:40.402 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:07.134122+0000 mon.a (mon.0) 74 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:10:40.402 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:07.134153+0000 mon.a (mon.0) 75 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:10:40.402 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:07.134167+0000 mon.a (mon.0) 76 : cluster 3 daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:10:40.402 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:07.134184+0000 mon.a (mon.0) 77 : cluster 3 daemon prometheus.a on smithi192 is in error state 2023-07-10T17:10:40.403 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:07.139795+0000 mon.a (mon.0) 78 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.403 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:07.147948+0000 mon.a (mon.0) 79 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.403 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:09.120105+0000 mgr.y (mgr.25390) 27 : cluster 0 pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.403 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:10.752894+0000 mon.a (mon.0) 80 : audit 0 from='client.34233 172.21.15.116:0/571250513' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:40.403 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:11.120975+0000 mgr.y (mgr.25390) 28 : cluster 0 pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.403 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:11.128461+0000 mon.a (mon.0) 81 : audit 1 from='client.34239 172.21.15.116:0/914235162' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: access denied 2023-07-10T17:10:40.403 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:13.122365+0000 mgr.y (mgr.25390) 29 : cluster 0 pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.403 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:14.083352+0000 mon.a (mon.0) 82 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:14.093126+0000 mon.a (mon.0) 83 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:14.890380+0000 mon.a (mon.0) 84 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:14.901650+0000 mon.a (mon.0) 85 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:15.123070+0000 mgr.y (mgr.25390) 30 : cluster 0 pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:17.123865+0000 mgr.y (mgr.25390) 31 : cluster 0 pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:19.125662+0000 mgr.y (mgr.25390) 32 : cluster 0 pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:21.126628+0000 mgr.y (mgr.25390) 33 : cluster 0 pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:23.128342+0000 mgr.y (mgr.25390) 34 : cluster 0 pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:25.129020+0000 mgr.y (mgr.25390) 35 : cluster 0 pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:25.274014+0000 mon.a (mon.0) 86 : audit 0 from='client.34257 172.21.15.116:0/218827705' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:40.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:25.454105+0000 mon.a (mon.0) 87 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:25.466964+0000 mon.a (mon.0) 88 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:25.468771+0000 mon.a (mon.0) 89 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:40.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:25.469941+0000 mon.a (mon.0) 90 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:10:40.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:10:25.470917+0000 mgr.y (mgr.25390) 36 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:40.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:25.471870+0000 mgr.y (mgr.25390) 37 : cluster 0 pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:25.472185+0000 mgr.y (mgr.25390) 38 : cluster 0 pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:25.477958+0000 mon.a (mon.0) 91 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:25.562669+0000 mon.a (mon.0) 92 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:10:40.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:25.566356+0000 mon.a (mon.0) 93 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:10:40.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:25.567937+0000 mon.a (mon.0) 94 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-10T17:10:40.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:25.568937+0000 mon.a (mon.0) 95 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2023-07-10T17:10:40.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:10:25.569879+0000 mgr.y (mgr.25390) 39 : cephadm 1 Upgrade: It appears safe to stop mon.b 2023-07-10T17:10:40.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:25.654742+0000 mon.a (mon.0) 96 : audit 1 from='client.34263 172.21.15.116:0/1098552669' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: access denied 2023-07-10T17:10:40.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:27.473786+0000 mgr.y (mgr.25390) 40 : cluster 0 pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2023-07-10T17:10:40.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:10:27.940293+0000 mgr.y (mgr.25390) 41 : cephadm 1 Upgrade: Updating mon.b 2023-07-10T17:10:40.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:27.951774+0000 mon.a (mon.0) 97 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:27.953582+0000 mon.a (mon.0) 98 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:10:40.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:27.955194+0000 mon.a (mon.0) 99 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:10:40.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:27.956667+0000 mon.a (mon.0) 100 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:40.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cephadm 2023-07-10T17:10:27.958349+0000 mgr.y (mgr.25390) 42 : cephadm 1 Deploying daemon mon.b on smithi192 2023-07-10T17:10:40.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:29.388890+0000 mgr.y (mgr.25390) 43 : audit 0 from='client.34269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:29.474342+0000 mgr.y (mgr.25390) 44 : cluster 0 pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2023-07-10T17:10:40.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:29.764306+0000 mgr.y (mgr.25390) 45 : audit 0 from='client.34275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:30.127159+0000 mgr.y (mgr.25390) 46 : audit 0 from='client.34281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:30.591093+0000 mon.c (mon.1) 2 : audit 0 from='client.44143 172.21.15.116:0/1142186097' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:10:40.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:34.000487+0000 mon.c (mon.1) 5 : cluster 1 mon.c calling monitor election 2023-07-10T17:10:40.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:34.004536+0000 mon.b (mon.2) 2 : cluster 1 mon.b calling monitor election 2023-07-10T17:10:40.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.007151+0000 mon.c (mon.1) 6 : cluster 1 mon.c calling monitor election 2023-07-10T17:10:40.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.008703+0000 mon.a (mon.0) 117 : cluster 0 fsmap 2023-07-10T17:10:40.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.008756+0000 mon.a (mon.0) 118 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2023-07-10T17:10:40.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.009379+0000 mon.a (mon.0) 119 : cluster 0 mgrmap e31: y(active, since 57s), standbys: x 2023-07-10T17:10:40.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.009662+0000 mon.a (mon.0) 120 : cluster 3 Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-10T17:10:40.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.012681+0000 mon.a (mon.0) 121 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:10:40.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.012706+0000 mon.a (mon.0) 122 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:10:40.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.012720+0000 mon.a (mon.0) 123 : cluster 3 daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:10:40.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.012740+0000 mon.a (mon.0) 124 : cluster 3 daemon prometheus.a on smithi192 is in error state 2023-07-10T17:10:40.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.012991+0000 mon.a (mon.0) 125 : cluster 1 mon.a calling monitor election 2023-07-10T17:10:40.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.029649+0000 mon.a (mon.0) 126 : cluster 1 mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-10T17:10:40.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.040038+0000 mon.a (mon.0) 127 : cluster 0 monmap e4: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0]} removed_ranks: {} 2023-07-10T17:10:40.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.040547+0000 mon.a (mon.0) 128 : cluster 0 fsmap 2023-07-10T17:10:40.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.040586+0000 mon.a (mon.0) 129 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2023-07-10T17:10:40.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.041086+0000 mon.a (mon.0) 130 : cluster 0 mgrmap e31: y(active, since 57s), standbys: x 2023-07-10T17:10:40.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.041288+0000 mon.a (mon.0) 131 : cluster 1 Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-10T17:10:40.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.049993+0000 mon.a (mon.0) 132 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:10:40.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.050038+0000 mon.a (mon.0) 133 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:10:40.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.050061+0000 mon.a (mon.0) 134 : cluster 3 daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:10:40.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.050086+0000 mon.a (mon.0) 135 : cluster 3 daemon prometheus.a on smithi192 is in error state 2023-07-10T17:10:40.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:39.054742+0000 mon.a (mon.0) 136 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:39.061733+0000 mon.a (mon.0) 137 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:39.629997+0000 mon.a (mon.0) 138 : audit 0 from='client.34317 172.21.15.116:0/607737785' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:40.412 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:40 smithi192 bash[50362]: audit 2023-07-10T17:10:40.013672+0000 mon.a (mon.0) 139 : audit 1 from='client.34323 172.21.15.116:0/3641116721' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: access denied 2023-07-10T17:10:40.412 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T16:54:21.709221+0000 mon.a (mon.0) 0 : 1 mkfs 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T17:10:40.412 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:06:42.160726+0000 mgr.x (mgr.24857) 25 : cephadm 1 Reconfiguring node-exporter.a (dependencies changed)... 2023-07-10T17:10:40.413 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:06:42.161197+0000 mgr.x (mgr.24857) 26 : cephadm 1 Reconfiguring daemon node-exporter.a on smithi116 2023-07-10T17:10:40.413 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:06:43.279509+0000 mgr.x (mgr.24857) 29 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2023-07-10T17:10:40.413 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:06:43.317778+0000 mgr.x (mgr.24857) 31 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi116 2023-07-10T17:10:40.413 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:06:52.953889+0000 mgr.x (mgr.24857) 38 : cephadm 1 Reconfiguring iscsi.foo.smithi116.fkthyl (dependencies changed)... 2023-07-10T17:10:40.413 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:06:52.969207+0000 mgr.x (mgr.24857) 39 : cephadm 1 Reconfiguring daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:10:40.413 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:06:55.793158+0000 mgr.x (mgr.24857) 41 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2023-07-10T17:10:40.413 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:06:56.037644+0000 mgr.x (mgr.24857) 43 : cephadm 1 Reconfiguring daemon prometheus.a on smithi192 2023-07-10T17:10:40.413 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:07:10.084390+0000 mgr.x (mgr.24857) 51 : cephadm 1 Reconfiguring node-exporter.b (dependencies changed)... 2023-07-10T17:10:40.414 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:07:10.085244+0000 mgr.x (mgr.24857) 52 : cephadm 1 Reconfiguring daemon node-exporter.b on smithi192 2023-07-10T17:10:40.414 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:07:11.754186+0000 mgr.x (mgr.24857) 53 : cephadm 1 Reconfiguring grafana.a (dependencies changed)... 2023-07-10T17:10:40.414 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:07:11.769017+0000 mgr.x (mgr.24857) 54 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2023-07-10T17:10:40.414 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:07:11.878848+0000 mgr.x (mgr.24857) 56 : cephadm 1 Reconfiguring daemon grafana.a on smithi192 2023-07-10T17:10:40.414 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:09.031472+0000 mgr.x (mgr.24857) 102 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:10:40.414 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:09.032983+0000 mgr.x (mgr.24857) 103 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:10:40.414 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:11.403617+0000 mgr.x (mgr.24857) 105 : cephadm 1 Upgrade: Updating mgr.y 2023-07-10T17:10:40.414 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:11.423499+0000 mgr.x (mgr.24857) 106 : cephadm 1 Deploying daemon mgr.y on smithi116 2023-07-10T17:10:40.415 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:32.705714+0000 mgr.x (mgr.24857) 122 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:40.415 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:32.810042+0000 mgr.x (mgr.24857) 125 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:10:40.415 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:32.810666+0000 mgr.x (mgr.24857) 126 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:10:40.415 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:32.813767+0000 mgr.x (mgr.24857) 127 : cephadm 1 Failing over to other MGR 2023-07-10T17:10:40.415 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:33.786136+0000 mgr.y (mgr.15252) 1 : cephadm 1 inventory: adjusted host smithi192 addr 'smithi192' -> '172.21.15.192' 2023-07-10T17:10:40.415 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:34.964152+0000 mgr.y (mgr.15252) 3 : cephadm 1 [10/Jul/2023:17:08:34] ENGINE Bus STARTING 2023-07-10T17:10:40.416 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T16:54:21.709221+0000 mon.a (mon.0) 0 : 1 mkfs 3f83c2ec-1f42-11ee-9b31-001a4aab830c 2023-07-10T17:10:40.416 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:06:42.160726+0000 mgr.x (mgr.24857) 25 : cephadm 1 Reconfiguring node-exporter.a (dependencies changed)... 2023-07-10T17:10:40.416 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:06:42.161197+0000 mgr.x (mgr.24857) 26 : cephadm 1 Reconfiguring daemon node-exporter.a on smithi116 2023-07-10T17:10:40.416 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:06:43.279509+0000 mgr.x (mgr.24857) 29 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2023-07-10T17:10:40.416 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:06:43.317778+0000 mgr.x (mgr.24857) 31 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi116 2023-07-10T17:10:40.416 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:06:52.953889+0000 mgr.x (mgr.24857) 38 : cephadm 1 Reconfiguring iscsi.foo.smithi116.fkthyl (dependencies changed)... 2023-07-10T17:10:40.417 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:06:52.969207+0000 mgr.x (mgr.24857) 39 : cephadm 1 Reconfiguring daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:10:40.417 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:06:55.793158+0000 mgr.x (mgr.24857) 41 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2023-07-10T17:10:40.417 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:06:56.037644+0000 mgr.x (mgr.24857) 43 : cephadm 1 Reconfiguring daemon prometheus.a on smithi192 2023-07-10T17:10:40.417 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:07:10.084390+0000 mgr.x (mgr.24857) 51 : cephadm 1 Reconfiguring node-exporter.b (dependencies changed)... 2023-07-10T17:10:40.417 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:07:10.085244+0000 mgr.x (mgr.24857) 52 : cephadm 1 Reconfiguring daemon node-exporter.b on smithi192 2023-07-10T17:10:40.417 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:07:11.754186+0000 mgr.x (mgr.24857) 53 : cephadm 1 Reconfiguring grafana.a (dependencies changed)... 2023-07-10T17:10:40.417 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:07:11.769017+0000 mgr.x (mgr.24857) 54 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2023-07-10T17:10:40.417 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:07:11.878848+0000 mgr.x (mgr.24857) 56 : cephadm 1 Reconfiguring daemon grafana.a on smithi192 2023-07-10T17:10:40.418 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:09.031472+0000 mgr.x (mgr.24857) 102 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:10:40.418 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:09.032983+0000 mgr.x (mgr.24857) 103 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:10:40.418 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:11.403617+0000 mgr.x (mgr.24857) 105 : cephadm 1 Upgrade: Updating mgr.y 2023-07-10T17:10:40.418 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:11.423499+0000 mgr.x (mgr.24857) 106 : cephadm 1 Deploying daemon mgr.y on smithi116 2023-07-10T17:10:40.418 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:32.705714+0000 mgr.x (mgr.24857) 122 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:40.418 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:32.810042+0000 mgr.x (mgr.24857) 125 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:10:40.418 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:32.810666+0000 mgr.x (mgr.24857) 126 : cephadm 1 Upgrade: Need to upgrade myself (mgr.x) 2023-07-10T17:10:40.418 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:32.813767+0000 mgr.x (mgr.24857) 127 : cephadm 1 Failing over to other MGR 2023-07-10T17:10:40.419 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:33.786136+0000 mgr.y (mgr.15252) 1 : cephadm 1 inventory: adjusted host smithi192 addr 'smithi192' -> '172.21.15.192' 2023-07-10T17:10:40.419 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:34.964152+0000 mgr.y (mgr.15252) 3 : cephadm 1 [10/Jul/2023:17:08:34] ENGINE Bus STARTING 2023-07-10T17:10:40.419 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:35.066231+0000 mgr.y (mgr.15252) 4 : cephadm 1 [10/Jul/2023:17:08:35] ENGINE Serving on http://172.21.15.116:8765 2023-07-10T17:10:40.419 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:35.193673+0000 mgr.y (mgr.15252) 5 : cephadm 1 [10/Jul/2023:17:08:35] ENGINE Serving on https://172.21.15.116:7150 2023-07-10T17:10:40.419 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:35.194073+0000 mgr.y (mgr.15252) 6 : cephadm 1 [10/Jul/2023:17:08:35] ENGINE Bus STARTED 2023-07-10T17:10:40.420 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:51.857316+0000 mgr.y (mgr.15252) 16 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:40.420 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:51.943607+0000 mgr.y (mgr.15252) 18 : cephadm 1 Reconfiguring iscsi.foo.smithi116.fkthyl (dependencies changed)... 2023-07-10T17:10:40.420 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:51.966478+0000 mgr.y (mgr.15252) 19 : cephadm 1 Reconfiguring daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:10:40.420 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:54.286766+0000 mgr.y (mgr.15252) 25 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2023-07-10T17:10:40.420 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:08:54.529934+0000 mgr.y (mgr.15252) 26 : cephadm 1 Reconfiguring daemon prometheus.a on smithi192 2023-07-10T17:10:40.420 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:09:01.831647+0000 mgr.y (mgr.15252) 33 : cephadm 1 Upgrade: Updating mgr.x 2023-07-10T17:10:40.420 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:09:01.841176+0000 mgr.y (mgr.15252) 34 : cephadm 1 Deploying daemon mgr.x on smithi192 2023-07-10T17:10:40.420 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:09:22.611122+0000 mgr.y (mgr.15252) 46 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:40.421 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:09:22.715387+0000 mgr.y (mgr.15252) 49 : cephadm 1 Upgrade: Setting container_image for all mgr 2023-07-10T17:10:40.421 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:09:22.751620+0000 mgr.y (mgr.15252) 50 : cephadm 1 Upgrade: It appears safe to stop mon.a 2023-07-10T17:10:40.421 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:09:26.328487+0000 mgr.y (mgr.15252) 56 : cephadm 1 Upgrade: Updating mon.a 2023-07-10T17:10:40.421 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:09:26.343243+0000 mgr.y (mgr.15252) 57 : cephadm 1 Deploying daemon mon.a on smithi116 2023-07-10T17:10:40.421 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:41.215841+0000 osd.0 (osd.0) 3 : cluster 3 failed to encode map e77 with expected crc 2023-07-10T17:10:40.421 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:41.217666+0000 osd.5 (osd.5) 3 : cluster 3 failed to encode map e77 with expected crc 2023-07-10T17:10:40.421 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:41.235415+0000 mon.a (mon.0) 34 : cluster 0 Standby manager daemon x started 2023-07-10T17:10:40.422 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:41.240959+0000 mon.a (mon.0) 35 : cluster 1 Manager daemon y is now available 2023-07-10T17:10:40.422 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:41.298769+0000 osd.7 (osd.7) 3 : cluster 3 failed to encode map e77 with expected crc 2023-07-10T17:10:40.422 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:41.334381+0000 osd.3 (osd.3) 3 : cluster 3 failed to encode map e77 with expected crc 2023-07-10T17:10:40.422 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:41.334961+0000 osd.4 (osd.4) 3 : cluster 3 failed to encode map e77 with expected crc 2023-07-10T17:10:40.422 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:09:41.744716+0000 mgr.y (mgr.25390) 1 : cephadm 1 [10/Jul/2023:17:09:41] ENGINE Bus STARTING 2023-07-10T17:10:40.422 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:09:41.847036+0000 mgr.y (mgr.25390) 2 : cephadm 1 [10/Jul/2023:17:09:41] ENGINE Serving on http://172.21.15.116:8765 2023-07-10T17:10:40.422 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:09:41.974125+0000 mgr.y (mgr.25390) 3 : cephadm 1 [10/Jul/2023:17:09:41] ENGINE Serving on https://172.21.15.116:7150 2023-07-10T17:10:40.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:35.066231+0000 mgr.y (mgr.15252) 4 : cephadm 1 [10/Jul/2023:17:08:35] ENGINE Serving on http://172.21.15.116:8765 2023-07-10T17:10:40.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:35.193673+0000 mgr.y (mgr.15252) 5 : cephadm 1 [10/Jul/2023:17:08:35] ENGINE Serving on https://172.21.15.116:7150 2023-07-10T17:10:40.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:35.194073+0000 mgr.y (mgr.15252) 6 : cephadm 1 [10/Jul/2023:17:08:35] ENGINE Bus STARTED 2023-07-10T17:10:40.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:51.857316+0000 mgr.y (mgr.15252) 16 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:40.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:51.943607+0000 mgr.y (mgr.15252) 18 : cephadm 1 Reconfiguring iscsi.foo.smithi116.fkthyl (dependencies changed)... 2023-07-10T17:10:40.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:51.966478+0000 mgr.y (mgr.15252) 19 : cephadm 1 Reconfiguring daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:10:40.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:54.286766+0000 mgr.y (mgr.15252) 25 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2023-07-10T17:10:40.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:08:54.529934+0000 mgr.y (mgr.15252) 26 : cephadm 1 Reconfiguring daemon prometheus.a on smithi192 2023-07-10T17:10:40.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:09:01.831647+0000 mgr.y (mgr.15252) 33 : cephadm 1 Upgrade: Updating mgr.x 2023-07-10T17:10:40.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:09:01.841176+0000 mgr.y (mgr.15252) 34 : cephadm 1 Deploying daemon mgr.x on smithi192 2023-07-10T17:10:40.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:09:22.611122+0000 mgr.y (mgr.15252) 46 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:40.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:09:22.715387+0000 mgr.y (mgr.15252) 49 : cephadm 1 Upgrade: Setting container_image for all mgr 2023-07-10T17:10:40.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:09:22.751620+0000 mgr.y (mgr.15252) 50 : cephadm 1 Upgrade: It appears safe to stop mon.a 2023-07-10T17:10:40.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:09:26.328487+0000 mgr.y (mgr.15252) 56 : cephadm 1 Upgrade: Updating mon.a 2023-07-10T17:10:40.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:09:26.343243+0000 mgr.y (mgr.15252) 57 : cephadm 1 Deploying daemon mon.a on smithi116 2023-07-10T17:10:40.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:41.215841+0000 osd.0 (osd.0) 3 : cluster 3 failed to encode map e77 with expected crc 2023-07-10T17:10:40.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:41.217666+0000 osd.5 (osd.5) 3 : cluster 3 failed to encode map e77 with expected crc 2023-07-10T17:10:40.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:41.235415+0000 mon.a (mon.0) 34 : cluster 0 Standby manager daemon x started 2023-07-10T17:10:40.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:41.240959+0000 mon.a (mon.0) 35 : cluster 1 Manager daemon y is now available 2023-07-10T17:10:40.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:41.298769+0000 osd.7 (osd.7) 3 : cluster 3 failed to encode map e77 with expected crc 2023-07-10T17:10:40.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:41.334381+0000 osd.3 (osd.3) 3 : cluster 3 failed to encode map e77 with expected crc 2023-07-10T17:10:40.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:41.334961+0000 osd.4 (osd.4) 3 : cluster 3 failed to encode map e77 with expected crc 2023-07-10T17:10:40.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:09:41.744716+0000 mgr.y (mgr.25390) 1 : cephadm 1 [10/Jul/2023:17:09:41] ENGINE Bus STARTING 2023-07-10T17:10:40.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:09:41.847036+0000 mgr.y (mgr.25390) 2 : cephadm 1 [10/Jul/2023:17:09:41] ENGINE Serving on http://172.21.15.116:8765 2023-07-10T17:10:40.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:09:41.974125+0000 mgr.y (mgr.25390) 3 : cephadm 1 [10/Jul/2023:17:09:41] ENGINE Serving on https://172.21.15.116:7150 2023-07-10T17:10:40.426 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:09:41.974317+0000 mgr.y (mgr.25390) 4 : cephadm 1 [10/Jul/2023:17:09:41] ENGINE Bus STARTED 2023-07-10T17:10:40.426 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:42.217941+0000 mon.a (mon.0) 40 : cluster 0 mgrmap e30: y(active, since 1.01682s), standbys: x 2023-07-10T17:10:40.427 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:42.231725+0000 mgr.y (mgr.25390) 5 : cluster 0 pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.427 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:43.210357+0000 mon.a (mon.0) 41 : cluster 1 Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2023-07-10T17:10:40.427 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:43.210394+0000 mon.a (mon.0) 42 : cluster 1 Cluster is now healthy 2023-07-10T17:10:40.427 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:43.231670+0000 mgr.y (mgr.25390) 6 : cluster 0 pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.427 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:43.492770+0000 mon.a (mon.0) 43 : cluster 0 mgrmap e31: y(active, since 2s), standbys: x 2023-07-10T17:10:40.427 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:45.232364+0000 mgr.y (mgr.25390) 7 : cluster 0 pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.427 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:47.233995+0000 mgr.y (mgr.25390) 8 : cluster 0 pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.428 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:49.192422+0000 mon.a (mon.0) 50 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.428 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:49.198955+0000 mon.a (mon.0) 51 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.428 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:49.234683+0000 mgr.y (mgr.25390) 9 : cluster 0 pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.428 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:51.235421+0000 mgr.y (mgr.25390) 10 : cluster 0 pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.428 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:53.237029+0000 mgr.y (mgr.25390) 11 : cluster 0 pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.428 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:55.237771+0000 mgr.y (mgr.25390) 12 : cluster 0 pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.428 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:56.426782+0000 mon.c (mon.1) 97 : audit 0 from='client.? 172.21.15.116:0/2649887320' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:40.428 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:56.797829+0000 mon.c (mon.1) 98 : audit 1 from='client.? 172.21.15.116:0/1164294302' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]: access denied 2023-07-10T17:10:40.429 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:56.981724+0000 mgr.y (mgr.25390) 13 : audit 0 from='client.25465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.429 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:57.239327+0000 mgr.y (mgr.25390) 14 : cluster 0 pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.429 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:57.363273+0000 mgr.y (mgr.25390) 15 : audit 0 from='client.25471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.429 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:57.744423+0000 mgr.y (mgr.25390) 16 : audit 0 from='client.25477 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.429 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:09:41.974317+0000 mgr.y (mgr.25390) 4 : cephadm 1 [10/Jul/2023:17:09:41] ENGINE Bus STARTED 2023-07-10T17:10:40.430 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:42.217941+0000 mon.a (mon.0) 40 : cluster 0 mgrmap e30: y(active, since 1.01682s), standbys: x 2023-07-10T17:10:40.430 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:42.231725+0000 mgr.y (mgr.25390) 5 : cluster 0 pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.430 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:43.210357+0000 mon.a (mon.0) 41 : cluster 1 Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2023-07-10T17:10:40.430 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:43.210394+0000 mon.a (mon.0) 42 : cluster 1 Cluster is now healthy 2023-07-10T17:10:40.430 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:43.231670+0000 mgr.y (mgr.25390) 6 : cluster 0 pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.430 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:43.492770+0000 mon.a (mon.0) 43 : cluster 0 mgrmap e31: y(active, since 2s), standbys: x 2023-07-10T17:10:40.430 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:45.232364+0000 mgr.y (mgr.25390) 7 : cluster 0 pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.430 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:47.233995+0000 mgr.y (mgr.25390) 8 : cluster 0 pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.431 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:49.192422+0000 mon.a (mon.0) 50 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.431 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:49.198955+0000 mon.a (mon.0) 51 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.431 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:49.234683+0000 mgr.y (mgr.25390) 9 : cluster 0 pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.431 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:51.235421+0000 mgr.y (mgr.25390) 10 : cluster 0 pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.431 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:53.237029+0000 mgr.y (mgr.25390) 11 : cluster 0 pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.431 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:55.237771+0000 mgr.y (mgr.25390) 12 : cluster 0 pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.431 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:56.426782+0000 mon.c (mon.1) 97 : audit 0 from='client.? 172.21.15.116:0/2649887320' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:40.432 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:56.797829+0000 mon.c (mon.1) 98 : audit 1 from='client.? 172.21.15.116:0/1164294302' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]: access denied 2023-07-10T17:10:40.432 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:56.981724+0000 mgr.y (mgr.25390) 13 : audit 0 from='client.25465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.432 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:57.239327+0000 mgr.y (mgr.25390) 14 : cluster 0 pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.432 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:57.363273+0000 mgr.y (mgr.25390) 15 : audit 0 from='client.25471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.433 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:57.744423+0000 mgr.y (mgr.25390) 16 : audit 0 from='client.25477 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.433 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:58.201338+0000 mon.a (mon.0) 52 : audit 0 from='client.34203 172.21.15.116:0/1768468329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:10:40.433 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:58.573745+0000 mgr.y (mgr.25390) 17 : audit 0 from='client.25489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.433 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:59.008117+0000 mon.c (mon.1) 99 : audit 0 from='client.? 172.21.15.116:0/3069563215' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:10:40.433 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:59.098231+0000 mon.a (mon.0) 53 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.434 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:59.108591+0000 mon.a (mon.0) 54 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.434 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:59.111286+0000 mon.a (mon.0) 55 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:40.434 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:59.112529+0000 mon.a (mon.0) 56 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:10:40.434 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:09:59.113529+0000 mgr.y (mgr.25390) 18 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:40.434 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:09:59.114638+0000 mgr.y (mgr.25390) 19 : cluster 0 pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:40.434 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:59.121473+0000 mon.a (mon.0) 57 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.434 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:59.208871+0000 mon.a (mon.0) 58 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:10:40.435 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:59.214394+0000 mon.a (mon.0) 59 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:10:40.435 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:59.216783+0000 mon.a (mon.0) 60 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-10T17:10:40.435 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:09:59.218328+0000 mon.a (mon.0) 61 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2023-07-10T17:10:40.435 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:09:59.219379+0000 mgr.y (mgr.25390) 20 : cephadm 1 Upgrade: It appears safe to stop mon.c 2023-07-10T17:10:40.435 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:00.000194+0000 mon.a (mon.0) 62 : cluster 1 overall HEALTH_OK 2023-07-10T17:10:40.435 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:00.105153+0000 mon.a (mon.0) 63 : cluster 3 Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:10:40.435 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:01.115429+0000 mgr.y (mgr.25390) 21 : cluster 0 pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:40.436 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:10:01.438560+0000 mgr.y (mgr.25390) 22 : cephadm 1 Upgrade: Updating mon.c 2023-07-10T17:10:40.436 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:01.448292+0000 mon.a (mon.0) 64 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.436 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:58.201338+0000 mon.a (mon.0) 52 : audit 0 from='client.34203 172.21.15.116:0/1768468329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:10:40.436 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:58.573745+0000 mgr.y (mgr.25390) 17 : audit 0 from='client.25489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.436 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:59.008117+0000 mon.c (mon.1) 99 : audit 0 from='client.? 172.21.15.116:0/3069563215' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:10:40.437 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:59.098231+0000 mon.a (mon.0) 53 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.437 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:59.108591+0000 mon.a (mon.0) 54 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.437 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:59.111286+0000 mon.a (mon.0) 55 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:40.437 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:59.112529+0000 mon.a (mon.0) 56 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:10:40.437 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:09:59.113529+0000 mgr.y (mgr.25390) 18 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:40.437 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:09:59.114638+0000 mgr.y (mgr.25390) 19 : cluster 0 pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:40.437 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:59.121473+0000 mon.a (mon.0) 57 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.437 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:59.208871+0000 mon.a (mon.0) 58 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:10:40.438 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:59.214394+0000 mon.a (mon.0) 59 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:10:40.438 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:59.216783+0000 mon.a (mon.0) 60 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-10T17:10:40.438 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:09:59.218328+0000 mon.a (mon.0) 61 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2023-07-10T17:10:40.438 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:09:59.219379+0000 mgr.y (mgr.25390) 20 : cephadm 1 Upgrade: It appears safe to stop mon.c 2023-07-10T17:10:40.438 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:00.000194+0000 mon.a (mon.0) 62 : cluster 1 overall HEALTH_OK 2023-07-10T17:10:40.438 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:00.105153+0000 mon.a (mon.0) 63 : cluster 3 Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:10:40.438 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:01.115429+0000 mgr.y (mgr.25390) 21 : cluster 0 pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:40.439 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:10:01.438560+0000 mgr.y (mgr.25390) 22 : cephadm 1 Upgrade: Updating mon.c 2023-07-10T17:10:40.439 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:01.448292+0000 mon.a (mon.0) 64 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.439 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:01.450164+0000 mon.a (mon.0) 65 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:10:40.439 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:01.452396+0000 mon.a (mon.0) 66 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:10:40.439 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:01.454067+0000 mon.a (mon.0) 67 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:40.439 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:10:01.455702+0000 mgr.y (mgr.25390) 23 : cephadm 1 Deploying daemon mon.c on smithi116 2023-07-10T17:10:40.440 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:03.117083+0000 mgr.y (mgr.25390) 24 : cluster 0 pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:40.440 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:06.833426+0000 mon.c (mon.1) 1 : cluster 1 mon.c calling monitor election 2023-07-10T17:10:40.440 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:07.109747+0000 mon.a (mon.0) 68 : cluster 1 mon.a calling monitor election 2023-07-10T17:10:40.440 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:07.113008+0000 mon.a (mon.0) 69 : cluster 1 mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-10T17:10:40.440 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:07.118390+0000 mgr.y (mgr.25390) 26 : cluster 0 pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:40.440 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:07.125193+0000 mon.a (mon.0) 70 : cluster 0 monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0]} removed_ranks: {} 2023-07-10T17:10:40.440 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:07.126355+0000 mon.a (mon.0) 71 : cluster 0 fsmap 2023-07-10T17:10:40.441 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:07.126413+0000 mon.a (mon.0) 72 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2023-07-10T17:10:40.441 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:07.127024+0000 mon.a (mon.0) 73 : cluster 0 mgrmap e31: y(active, since 25s), standbys: x 2023-07-10T17:10:40.441 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:07.134122+0000 mon.a (mon.0) 74 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:10:40.441 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:07.134153+0000 mon.a (mon.0) 75 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:10:40.441 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:07.134167+0000 mon.a (mon.0) 76 : cluster 3 daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:10:40.441 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:07.134184+0000 mon.a (mon.0) 77 : cluster 3 daemon prometheus.a on smithi192 is in error state 2023-07-10T17:10:40.441 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:07.139795+0000 mon.a (mon.0) 78 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.441 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:07.147948+0000 mon.a (mon.0) 79 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.442 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:09.120105+0000 mgr.y (mgr.25390) 27 : cluster 0 pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.442 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:10.752894+0000 mon.a (mon.0) 80 : audit 0 from='client.34233 172.21.15.116:0/571250513' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:40.442 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:11.120975+0000 mgr.y (mgr.25390) 28 : cluster 0 pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.442 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:01.450164+0000 mon.a (mon.0) 65 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:10:40.442 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:01.452396+0000 mon.a (mon.0) 66 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:10:40.442 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:01.454067+0000 mon.a (mon.0) 67 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:40.443 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:10:01.455702+0000 mgr.y (mgr.25390) 23 : cephadm 1 Deploying daemon mon.c on smithi116 2023-07-10T17:10:40.443 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:03.117083+0000 mgr.y (mgr.25390) 24 : cluster 0 pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:40.443 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:06.833426+0000 mon.c (mon.1) 1 : cluster 1 mon.c calling monitor election 2023-07-10T17:10:40.443 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:07.109747+0000 mon.a (mon.0) 68 : cluster 1 mon.a calling monitor election 2023-07-10T17:10:40.443 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:07.113008+0000 mon.a (mon.0) 69 : cluster 1 mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-10T17:10:40.443 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:07.118390+0000 mgr.y (mgr.25390) 26 : cluster 0 pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-07-10T17:10:40.443 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:07.125193+0000 mon.a (mon.0) 70 : cluster 0 monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0]} removed_ranks: {} 2023-07-10T17:10:40.444 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:07.126355+0000 mon.a (mon.0) 71 : cluster 0 fsmap 2023-07-10T17:10:40.444 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:07.126413+0000 mon.a (mon.0) 72 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2023-07-10T17:10:40.444 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:07.127024+0000 mon.a (mon.0) 73 : cluster 0 mgrmap e31: y(active, since 25s), standbys: x 2023-07-10T17:10:40.444 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:07.134122+0000 mon.a (mon.0) 74 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:10:40.444 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:07.134153+0000 mon.a (mon.0) 75 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:10:40.444 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:07.134167+0000 mon.a (mon.0) 76 : cluster 3 daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:10:40.444 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:07.134184+0000 mon.a (mon.0) 77 : cluster 3 daemon prometheus.a on smithi192 is in error state 2023-07-10T17:10:40.444 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:07.139795+0000 mon.a (mon.0) 78 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.445 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:07.147948+0000 mon.a (mon.0) 79 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.445 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:09.120105+0000 mgr.y (mgr.25390) 27 : cluster 0 pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.445 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:10.752894+0000 mon.a (mon.0) 80 : audit 0 from='client.34233 172.21.15.116:0/571250513' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:40.445 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:11.120975+0000 mgr.y (mgr.25390) 28 : cluster 0 pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.445 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:11.128461+0000 mon.a (mon.0) 81 : audit 1 from='client.34239 172.21.15.116:0/914235162' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: access denied 2023-07-10T17:10:40.445 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:13.122365+0000 mgr.y (mgr.25390) 29 : cluster 0 pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.446 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:14.083352+0000 mon.a (mon.0) 82 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.446 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:14.093126+0000 mon.a (mon.0) 83 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.446 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:14.890380+0000 mon.a (mon.0) 84 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.446 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:14.901650+0000 mon.a (mon.0) 85 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.446 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:15.123070+0000 mgr.y (mgr.25390) 30 : cluster 0 pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.446 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:17.123865+0000 mgr.y (mgr.25390) 31 : cluster 0 pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.446 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:19.125662+0000 mgr.y (mgr.25390) 32 : cluster 0 pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.447 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:21.126628+0000 mgr.y (mgr.25390) 33 : cluster 0 pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.447 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:23.128342+0000 mgr.y (mgr.25390) 34 : cluster 0 pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.447 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:25.129020+0000 mgr.y (mgr.25390) 35 : cluster 0 pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.447 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:25.274014+0000 mon.a (mon.0) 86 : audit 0 from='client.34257 172.21.15.116:0/218827705' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:40.447 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:25.454105+0000 mon.a (mon.0) 87 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.447 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:25.466964+0000 mon.a (mon.0) 88 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.447 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:25.468771+0000 mon.a (mon.0) 89 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:40.447 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:25.469941+0000 mon.a (mon.0) 90 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:10:40.448 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:10:25.470917+0000 mgr.y (mgr.25390) 36 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:40.448 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:25.471870+0000 mgr.y (mgr.25390) 37 : cluster 0 pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.448 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:25.472185+0000 mgr.y (mgr.25390) 38 : cluster 0 pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.448 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:25.477958+0000 mon.a (mon.0) 91 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.448 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:11.128461+0000 mon.a (mon.0) 81 : audit 1 from='client.34239 172.21.15.116:0/914235162' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: access denied 2023-07-10T17:10:40.449 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:13.122365+0000 mgr.y (mgr.25390) 29 : cluster 0 pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.449 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:14.083352+0000 mon.a (mon.0) 82 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.449 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:14.093126+0000 mon.a (mon.0) 83 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.449 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:14.890380+0000 mon.a (mon.0) 84 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.449 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:14.901650+0000 mon.a (mon.0) 85 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.449 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:15.123070+0000 mgr.y (mgr.25390) 30 : cluster 0 pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.449 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:17.123865+0000 mgr.y (mgr.25390) 31 : cluster 0 pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.449 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:19.125662+0000 mgr.y (mgr.25390) 32 : cluster 0 pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.450 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:21.126628+0000 mgr.y (mgr.25390) 33 : cluster 0 pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.450 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:23.128342+0000 mgr.y (mgr.25390) 34 : cluster 0 pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:40.450 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:25.129020+0000 mgr.y (mgr.25390) 35 : cluster 0 pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.450 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:25.274014+0000 mon.a (mon.0) 86 : audit 0 from='client.34257 172.21.15.116:0/218827705' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:40.450 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:25.454105+0000 mon.a (mon.0) 87 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.450 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:25.466964+0000 mon.a (mon.0) 88 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.450 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:25.468771+0000 mon.a (mon.0) 89 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:40.451 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:25.469941+0000 mon.a (mon.0) 90 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:10:40.451 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:10:25.470917+0000 mgr.y (mgr.25390) 36 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:40.451 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:25.471870+0000 mgr.y (mgr.25390) 37 : cluster 0 pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.451 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:25.472185+0000 mgr.y (mgr.25390) 38 : cluster 0 pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:40.451 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:25.477958+0000 mon.a (mon.0) 91 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.451 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:25.562669+0000 mon.a (mon.0) 92 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:10:40.452 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:25.566356+0000 mon.a (mon.0) 93 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:10:40.452 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:25.567937+0000 mon.a (mon.0) 94 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-10T17:10:40.452 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:25.568937+0000 mon.a (mon.0) 95 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2023-07-10T17:10:40.452 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:10:25.569879+0000 mgr.y (mgr.25390) 39 : cephadm 1 Upgrade: It appears safe to stop mon.b 2023-07-10T17:10:40.452 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:25.654742+0000 mon.a (mon.0) 96 : audit 1 from='client.34263 172.21.15.116:0/1098552669' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: access denied 2023-07-10T17:10:40.452 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:27.473786+0000 mgr.y (mgr.25390) 40 : cluster 0 pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2023-07-10T17:10:40.452 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:10:27.940293+0000 mgr.y (mgr.25390) 41 : cephadm 1 Upgrade: Updating mon.b 2023-07-10T17:10:40.453 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:27.951774+0000 mon.a (mon.0) 97 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.453 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:27.953582+0000 mon.a (mon.0) 98 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:10:40.453 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:27.955194+0000 mon.a (mon.0) 99 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:10:40.453 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:27.956667+0000 mon.a (mon.0) 100 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:40.453 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cephadm 2023-07-10T17:10:27.958349+0000 mgr.y (mgr.25390) 42 : cephadm 1 Deploying daemon mon.b on smithi192 2023-07-10T17:10:40.453 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:29.388890+0000 mgr.y (mgr.25390) 43 : audit 0 from='client.34269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.453 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:29.474342+0000 mgr.y (mgr.25390) 44 : cluster 0 pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2023-07-10T17:10:40.453 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:29.764306+0000 mgr.y (mgr.25390) 45 : audit 0 from='client.34275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.454 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:30.127159+0000 mgr.y (mgr.25390) 46 : audit 0 from='client.34281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.454 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:30.591093+0000 mon.c (mon.1) 2 : audit 0 from='client.44143 172.21.15.116:0/1142186097' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:10:40.454 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:34.000487+0000 mon.c (mon.1) 5 : cluster 1 mon.c calling monitor election 2023-07-10T17:10:40.454 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:34.004536+0000 mon.b (mon.2) 2 : cluster 1 mon.b calling monitor election 2023-07-10T17:10:40.454 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:25.562669+0000 mon.a (mon.0) 92 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:10:40.454 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:25.566356+0000 mon.a (mon.0) 93 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:10:40.455 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:25.567937+0000 mon.a (mon.0) 94 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-07-10T17:10:40.455 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:25.568937+0000 mon.a (mon.0) 95 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2023-07-10T17:10:40.455 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:10:25.569879+0000 mgr.y (mgr.25390) 39 : cephadm 1 Upgrade: It appears safe to stop mon.b 2023-07-10T17:10:40.455 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:25.654742+0000 mon.a (mon.0) 96 : audit 1 from='client.34263 172.21.15.116:0/1098552669' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: access denied 2023-07-10T17:10:40.455 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:27.473786+0000 mgr.y (mgr.25390) 40 : cluster 0 pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2023-07-10T17:10:40.455 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:10:27.940293+0000 mgr.y (mgr.25390) 41 : cephadm 1 Upgrade: Updating mon.b 2023-07-10T17:10:40.455 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:27.951774+0000 mon.a (mon.0) 97 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.456 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:27.953582+0000 mon.a (mon.0) 98 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:10:40.456 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:27.955194+0000 mon.a (mon.0) 99 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:10:40.456 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:27.956667+0000 mon.a (mon.0) 100 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:40.456 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cephadm 2023-07-10T17:10:27.958349+0000 mgr.y (mgr.25390) 42 : cephadm 1 Deploying daemon mon.b on smithi192 2023-07-10T17:10:40.456 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:29.388890+0000 mgr.y (mgr.25390) 43 : audit 0 from='client.34269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.456 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:29.474342+0000 mgr.y (mgr.25390) 44 : cluster 0 pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2023-07-10T17:10:40.456 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:29.764306+0000 mgr.y (mgr.25390) 45 : audit 0 from='client.34275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.456 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:30.127159+0000 mgr.y (mgr.25390) 46 : audit 0 from='client.34281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:10:40.457 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:30.591093+0000 mon.c (mon.1) 2 : audit 0 from='client.44143 172.21.15.116:0/1142186097' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:10:40.457 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:34.000487+0000 mon.c (mon.1) 5 : cluster 1 mon.c calling monitor election 2023-07-10T17:10:40.457 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:34.004536+0000 mon.b (mon.2) 2 : cluster 1 mon.b calling monitor election 2023-07-10T17:10:40.457 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.007151+0000 mon.c (mon.1) 6 : cluster 1 mon.c calling monitor election 2023-07-10T17:10:40.457 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.008703+0000 mon.a (mon.0) 117 : cluster 0 fsmap 2023-07-10T17:10:40.458 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.008756+0000 mon.a (mon.0) 118 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2023-07-10T17:10:40.458 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.009379+0000 mon.a (mon.0) 119 : cluster 0 mgrmap e31: y(active, since 57s), standbys: x 2023-07-10T17:10:40.458 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.009662+0000 mon.a (mon.0) 120 : cluster 3 Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-10T17:10:40.458 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.012681+0000 mon.a (mon.0) 121 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:10:40.458 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.012706+0000 mon.a (mon.0) 122 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:10:40.458 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.012720+0000 mon.a (mon.0) 123 : cluster 3 daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:10:40.458 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.012740+0000 mon.a (mon.0) 124 : cluster 3 daemon prometheus.a on smithi192 is in error state 2023-07-10T17:10:40.458 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.012991+0000 mon.a (mon.0) 125 : cluster 1 mon.a calling monitor election 2023-07-10T17:10:40.459 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.029649+0000 mon.a (mon.0) 126 : cluster 1 mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-10T17:10:40.459 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.040038+0000 mon.a (mon.0) 127 : cluster 0 monmap e4: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0]} removed_ranks: {} 2023-07-10T17:10:40.459 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.040547+0000 mon.a (mon.0) 128 : cluster 0 fsmap 2023-07-10T17:10:40.459 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.040586+0000 mon.a (mon.0) 129 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2023-07-10T17:10:40.459 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.041086+0000 mon.a (mon.0) 130 : cluster 0 mgrmap e31: y(active, since 57s), standbys: x 2023-07-10T17:10:40.459 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.041288+0000 mon.a (mon.0) 131 : cluster 1 Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-10T17:10:40.459 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.049993+0000 mon.a (mon.0) 132 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:10:40.460 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.050038+0000 mon.a (mon.0) 133 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:10:40.460 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.050061+0000 mon.a (mon.0) 134 : cluster 3 daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:10:40.460 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.050086+0000 mon.a (mon.0) 135 : cluster 3 daemon prometheus.a on smithi192 is in error state 2023-07-10T17:10:40.460 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:39.054742+0000 mon.a (mon.0) 136 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.460 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:39.061733+0000 mon.a (mon.0) 137 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.460 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:39.629997+0000 mon.a (mon.0) 138 : audit 0 from='client.34317 172.21.15.116:0/607737785' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:40.460 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[65980]: audit 2023-07-10T17:10:40.013672+0000 mon.a (mon.0) 139 : audit 1 from='client.34323 172.21.15.116:0/3641116721' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: access denied 2023-07-10T17:10:40.461 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.007151+0000 mon.c (mon.1) 6 : cluster 1 mon.c calling monitor election 2023-07-10T17:10:40.461 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.008703+0000 mon.a (mon.0) 117 : cluster 0 fsmap 2023-07-10T17:10:40.461 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.008756+0000 mon.a (mon.0) 118 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2023-07-10T17:10:40.461 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.009379+0000 mon.a (mon.0) 119 : cluster 0 mgrmap e31: y(active, since 57s), standbys: x 2023-07-10T17:10:40.461 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.009662+0000 mon.a (mon.0) 120 : cluster 3 Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-10T17:10:40.461 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.012681+0000 mon.a (mon.0) 121 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:10:40.461 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.012706+0000 mon.a (mon.0) 122 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:10:40.462 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.012720+0000 mon.a (mon.0) 123 : cluster 3 daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:10:40.462 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.012740+0000 mon.a (mon.0) 124 : cluster 3 daemon prometheus.a on smithi192 is in error state 2023-07-10T17:10:40.462 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.012991+0000 mon.a (mon.0) 125 : cluster 1 mon.a calling monitor election 2023-07-10T17:10:40.462 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.029649+0000 mon.a (mon.0) 126 : cluster 1 mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-10T17:10:40.462 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.040038+0000 mon.a (mon.0) 127 : cluster 0 monmap e4: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0]} removed_ranks: {} 2023-07-10T17:10:40.462 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.040547+0000 mon.a (mon.0) 128 : cluster 0 fsmap 2023-07-10T17:10:40.462 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.040586+0000 mon.a (mon.0) 129 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2023-07-10T17:10:40.462 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.041086+0000 mon.a (mon.0) 130 : cluster 0 mgrmap e31: y(active, since 57s), standbys: x 2023-07-10T17:10:40.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.041288+0000 mon.a (mon.0) 131 : cluster 1 Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-10T17:10:40.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.049993+0000 mon.a (mon.0) 132 : cluster 3 Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:10:40.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.050038+0000 mon.a (mon.0) 133 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:10:40.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.050061+0000 mon.a (mon.0) 134 : cluster 3 daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:10:40.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.050086+0000 mon.a (mon.0) 135 : cluster 3 daemon prometheus.a on smithi192 is in error state 2023-07-10T17:10:40.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:39.054742+0000 mon.a (mon.0) 136 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:39.061733+0000 mon.a (mon.0) 137 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:40.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:39.629997+0000 mon.a (mon.0) 138 : audit 0 from='client.34317 172.21.15.116:0/607737785' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:40.464 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:40 smithi116 bash[63900]: audit 2023-07-10T17:10:40.013672+0000 mon.a (mon.0) 139 : audit 1 from='client.34323 172.21.15.116:0/3641116721' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: access denied 2023-07-10T17:10:41.085 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:41 smithi116 bash[65980]: cluster 2023-07-10T17:10:39.479631+0000 mgr.y (mgr.25390) 52 : cluster 0 pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:41.085 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:41 smithi116 bash[63900]: cluster 2023-07-10T17:10:39.479631+0000 mgr.y (mgr.25390) 52 : cluster 0 pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:41.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:41 smithi192 bash[50362]: cluster 2023-07-10T17:10:39.479631+0000 mgr.y (mgr.25390) 52 : cluster 0 pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:43.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:43 smithi192 bash[50362]: cluster 2023-07-10T17:10:41.480483+0000 mgr.y (mgr.25390) 53 : cluster 0 pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:43.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:43 smithi116 bash[65980]: cluster 2023-07-10T17:10:41.480483+0000 mgr.y (mgr.25390) 53 : cluster 0 pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:43.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:43 smithi116 bash[63900]: cluster 2023-07-10T17:10:41.480483+0000 mgr.y (mgr.25390) 53 : cluster 0 pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:45.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:45 smithi192 bash[50362]: cluster 2023-07-10T17:10:43.482434+0000 mgr.y (mgr.25390) 54 : cluster 0 pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:45.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:45 smithi116 bash[65980]: cluster 2023-07-10T17:10:43.482434+0000 mgr.y (mgr.25390) 54 : cluster 0 pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:45.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:45 smithi116 bash[63900]: cluster 2023-07-10T17:10:43.482434+0000 mgr.y (mgr.25390) 54 : cluster 0 pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:46.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:46 smithi192 bash[50362]: cluster 2023-07-10T17:10:45.483335+0000 mgr.y (mgr.25390) 55 : cluster 0 pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:46.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:46 smithi192 bash[50362]: audit 2023-07-10T17:10:45.610370+0000 mon.a (mon.0) 140 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:46.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:46 smithi192 bash[50362]: audit 2023-07-10T17:10:45.621665+0000 mon.a (mon.0) 141 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:46.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:46 smithi192 bash[50362]: audit 2023-07-10T17:10:46.421082+0000 mon.a (mon.0) 142 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:46.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:46 smithi192 bash[50362]: audit 2023-07-10T17:10:46.431774+0000 mon.a (mon.0) 143 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:46.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:46 smithi116 bash[65980]: cluster 2023-07-10T17:10:45.483335+0000 mgr.y (mgr.25390) 55 : cluster 0 pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:46.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:46 smithi116 bash[65980]: audit 2023-07-10T17:10:45.610370+0000 mon.a (mon.0) 140 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:46.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:46 smithi116 bash[65980]: audit 2023-07-10T17:10:45.621665+0000 mon.a (mon.0) 141 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:46.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:46 smithi116 bash[65980]: audit 2023-07-10T17:10:46.421082+0000 mon.a (mon.0) 142 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:46.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:46 smithi116 bash[65980]: audit 2023-07-10T17:10:46.431774+0000 mon.a (mon.0) 143 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:46.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:46 smithi116 bash[63900]: cluster 2023-07-10T17:10:45.483335+0000 mgr.y (mgr.25390) 55 : cluster 0 pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:46.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:46 smithi116 bash[63900]: audit 2023-07-10T17:10:45.610370+0000 mon.a (mon.0) 140 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:46.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:46 smithi116 bash[63900]: audit 2023-07-10T17:10:45.621665+0000 mon.a (mon.0) 141 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:46.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:46 smithi116 bash[63900]: audit 2023-07-10T17:10:46.421082+0000 mon.a (mon.0) 142 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:46.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:46 smithi116 bash[63900]: audit 2023-07-10T17:10:46.431774+0000 mon.a (mon.0) 143 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:48.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:48 smithi192 bash[50362]: cluster 2023-07-10T17:10:47.484943+0000 mgr.y (mgr.25390) 56 : cluster 0 pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:48.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:48 smithi116 bash[65980]: cluster 2023-07-10T17:10:47.484943+0000 mgr.y (mgr.25390) 56 : cluster 0 pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:48.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:48 smithi116 bash[63900]: cluster 2023-07-10T17:10:47.484943+0000 mgr.y (mgr.25390) 56 : cluster 0 pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:50.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:50 smithi192 bash[50362]: cluster 2023-07-10T17:10:49.485721+0000 mgr.y (mgr.25390) 57 : cluster 0 pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:50.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:50 smithi116 bash[63900]: cluster 2023-07-10T17:10:49.485721+0000 mgr.y (mgr.25390) 57 : cluster 0 pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:50.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:50 smithi116 bash[65980]: cluster 2023-07-10T17:10:49.485721+0000 mgr.y (mgr.25390) 57 : cluster 0 pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:52.729 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:52 smithi192 bash[50362]: cluster 2023-07-10T17:10:51.486472+0000 mgr.y (mgr.25390) 58 : cluster 0 pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:52.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:52 smithi116 bash[63900]: cluster 2023-07-10T17:10:51.486472+0000 mgr.y (mgr.25390) 58 : cluster 0 pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:52.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:52 smithi116 bash[65980]: cluster 2023-07-10T17:10:51.486472+0000 mgr.y (mgr.25390) 58 : cluster 0 pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:54.887 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:54 smithi116 bash[63900]: cluster 2023-07-10T17:10:53.488215+0000 mgr.y (mgr.25390) 59 : cluster 0 pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:54.887 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:54 smithi116 bash[63900]: audit 2023-07-10T17:10:54.490923+0000 mon.c (mon.1) 7 : audit 0 from='client.34341 172.21.15.116:0/2941332148' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:54.887 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:54 smithi116 bash[65980]: cluster 2023-07-10T17:10:53.488215+0000 mgr.y (mgr.25390) 59 : cluster 0 pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:54.888 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:54 smithi116 bash[65980]: audit 2023-07-10T17:10:54.490923+0000 mon.c (mon.1) 7 : audit 0 from='client.34341 172.21.15.116:0/2941332148' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:54.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:54 smithi192 bash[50362]: cluster 2023-07-10T17:10:53.488215+0000 mgr.y (mgr.25390) 59 : cluster 0 pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:10:54.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:54 smithi192 bash[50362]: audit 2023-07-10T17:10:54.490923+0000 mon.c (mon.1) 7 : audit 0 from='client.34341 172.21.15.116:0/2941332148' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:10:55.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:55 smithi116 bash[63900]: audit 2023-07-10T17:10:54.875263+0000 mon.c (mon.1) 8 : audit 1 from='client.34347 172.21.15.116:0/28636037' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]: access denied 2023-07-10T17:10:55.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:55 smithi116 bash[65980]: audit 2023-07-10T17:10:54.875263+0000 mon.c (mon.1) 8 : audit 1 from='client.34347 172.21.15.116:0/28636037' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]: access denied 2023-07-10T17:10:56.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:55 smithi192 bash[50362]: audit 2023-07-10T17:10:54.875263+0000 mon.c (mon.1) 8 : audit 1 from='client.34347 172.21.15.116:0/28636037' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]: access denied 2023-07-10T17:10:56.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[63900]: cluster 2023-07-10T17:10:55.489002+0000 mgr.y (mgr.25390) 60 : cluster 0 pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:56.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[63900]: audit 2023-07-10T17:10:56.384862+0000 mon.a (mon.0) 144 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:56.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[63900]: audit 2023-07-10T17:10:56.394625+0000 mon.a (mon.0) 145 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:56.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[63900]: audit 2023-07-10T17:10:56.396399+0000 mon.a (mon.0) 146 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:56.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[63900]: audit 2023-07-10T17:10:56.397931+0000 mon.a (mon.0) 147 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:10:56.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[63900]: audit 2023-07-10T17:10:56.407410+0000 mon.a (mon.0) 148 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:56.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[63900]: audit 2023-07-10T17:10:56.480874+0000 mon.a (mon.0) 149 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:10:56.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[63900]: audit 2023-07-10T17:10:56.482743+0000 mon.a (mon.0) 150 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:10:56.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[63900]: audit 2023-07-10T17:10:56.484525+0000 mon.a (mon.0) 151 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:56.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[65980]: cluster 2023-07-10T17:10:55.489002+0000 mgr.y (mgr.25390) 60 : cluster 0 pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:56.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[65980]: audit 2023-07-10T17:10:56.384862+0000 mon.a (mon.0) 144 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:56.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[65980]: audit 2023-07-10T17:10:56.394625+0000 mon.a (mon.0) 145 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:56.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[65980]: audit 2023-07-10T17:10:56.396399+0000 mon.a (mon.0) 146 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:56.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[65980]: audit 2023-07-10T17:10:56.397931+0000 mon.a (mon.0) 147 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:10:56.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[65980]: audit 2023-07-10T17:10:56.407410+0000 mon.a (mon.0) 148 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:56.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[65980]: audit 2023-07-10T17:10:56.480874+0000 mon.a (mon.0) 149 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:10:56.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[65980]: audit 2023-07-10T17:10:56.482743+0000 mon.a (mon.0) 150 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:10:56.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:56 smithi116 bash[65980]: audit 2023-07-10T17:10:56.484525+0000 mon.a (mon.0) 151 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:57.141 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:56 smithi192 bash[50362]: cluster 2023-07-10T17:10:55.489002+0000 mgr.y (mgr.25390) 60 : cluster 0 pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:57.142 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:56 smithi192 bash[50362]: audit 2023-07-10T17:10:56.384862+0000 mon.a (mon.0) 144 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:57.142 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:56 smithi192 bash[50362]: audit 2023-07-10T17:10:56.394625+0000 mon.a (mon.0) 145 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:57.143 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:56 smithi192 bash[50362]: audit 2023-07-10T17:10:56.396399+0000 mon.a (mon.0) 146 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:57.143 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:56 smithi192 bash[50362]: audit 2023-07-10T17:10:56.397931+0000 mon.a (mon.0) 147 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:10:57.144 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:56 smithi192 bash[50362]: audit 2023-07-10T17:10:56.407410+0000 mon.a (mon.0) 148 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:10:57.144 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:56 smithi192 bash[50362]: audit 2023-07-10T17:10:56.480874+0000 mon.a (mon.0) 149 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:10:57.144 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:56 smithi192 bash[50362]: audit 2023-07-10T17:10:56.482743+0000 mon.a (mon.0) 150 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:10:57.144 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:56 smithi192 bash[50362]: audit 2023-07-10T17:10:56.484525+0000 mon.a (mon.0) 151 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:10:57.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:57 smithi116 bash[63900]: cephadm 2023-07-10T17:10:56.399423+0000 mgr.y (mgr.25390) 61 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:57.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:57 smithi116 bash[63900]: cluster 2023-07-10T17:10:56.400455+0000 mgr.y (mgr.25390) 62 : cluster 0 pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:57.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:57 smithi116 bash[63900]: cluster 2023-07-10T17:10:56.402402+0000 mgr.y (mgr.25390) 63 : cluster 0 pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:57.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:57 smithi116 bash[63900]: cephadm 2023-07-10T17:10:56.480036+0000 mgr.y (mgr.25390) 64 : cephadm 1 Reconfiguring mon.a (monmap changed)... 2023-07-10T17:10:57.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:57 smithi116 bash[63900]: cephadm 2023-07-10T17:10:56.486294+0000 mgr.y (mgr.25390) 65 : cephadm 1 Reconfiguring daemon mon.a on smithi116 2023-07-10T17:10:57.957 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:57 smithi116 bash[65980]: cephadm 2023-07-10T17:10:56.399423+0000 mgr.y (mgr.25390) 61 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:57.957 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:57 smithi116 bash[65980]: cluster 2023-07-10T17:10:56.400455+0000 mgr.y (mgr.25390) 62 : cluster 0 pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:57.957 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:57 smithi116 bash[65980]: cluster 2023-07-10T17:10:56.402402+0000 mgr.y (mgr.25390) 63 : cluster 0 pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:57.958 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:57 smithi116 bash[65980]: cephadm 2023-07-10T17:10:56.480036+0000 mgr.y (mgr.25390) 64 : cephadm 1 Reconfiguring mon.a (monmap changed)... 2023-07-10T17:10:57.958 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:57 smithi116 bash[65980]: cephadm 2023-07-10T17:10:56.486294+0000 mgr.y (mgr.25390) 65 : cephadm 1 Reconfiguring daemon mon.a on smithi116 2023-07-10T17:10:58.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:57 smithi192 bash[50362]: cephadm 2023-07-10T17:10:56.399423+0000 mgr.y (mgr.25390) 61 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:10:58.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:57 smithi192 bash[50362]: cluster 2023-07-10T17:10:56.400455+0000 mgr.y (mgr.25390) 62 : cluster 0 pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:58.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:57 smithi192 bash[50362]: cluster 2023-07-10T17:10:56.402402+0000 mgr.y (mgr.25390) 63 : cluster 0 pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:10:58.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:57 smithi192 bash[50362]: cephadm 2023-07-10T17:10:56.480036+0000 mgr.y (mgr.25390) 64 : cephadm 1 Reconfiguring mon.a (monmap changed)... 2023-07-10T17:10:58.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:57 smithi192 bash[50362]: cephadm 2023-07-10T17:10:56.486294+0000 mgr.y (mgr.25390) 65 : cephadm 1 Reconfiguring daemon mon.a on smithi116 2023-07-10T17:11:00.044 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:59 smithi116 bash[63900]: cluster 2023-07-10T17:10:58.404128+0000 mgr.y (mgr.25390) 66 : cluster 0 pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-07-10T17:11:00.045 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:59 smithi116 bash[63900]: audit 2023-07-10T17:10:58.728676+0000 mon.a (mon.0) 152 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:00.045 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:59 smithi116 bash[63900]: audit 2023-07-10T17:10:58.737826+0000 mon.a (mon.0) 153 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:00.045 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:59 smithi116 bash[63900]: cephadm 2023-07-10T17:10:58.755773+0000 mgr.y (mgr.25390) 67 : cephadm 1 Reconfiguring mgr.y (monmap changed)... 2023-07-10T17:11:00.045 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:59 smithi116 bash[63900]: audit 2023-07-10T17:10:58.756466+0000 mon.a (mon.0) 154 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:11:00.045 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:59 smithi116 bash[63900]: audit 2023-07-10T17:10:58.758308+0000 mon.a (mon.0) 155 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T17:11:00.045 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:59 smithi116 bash[63900]: audit 2023-07-10T17:10:58.760219+0000 mon.a (mon.0) 156 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:00.045 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:10:59 smithi116 bash[63900]: cephadm 2023-07-10T17:10:58.762154+0000 mgr.y (mgr.25390) 68 : cephadm 1 Reconfiguring daemon mgr.y on smithi116 2023-07-10T17:11:00.046 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:59 smithi116 bash[65980]: cluster 2023-07-10T17:10:58.404128+0000 mgr.y (mgr.25390) 66 : cluster 0 pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-07-10T17:11:00.046 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:59 smithi116 bash[65980]: audit 2023-07-10T17:10:58.728676+0000 mon.a (mon.0) 152 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:00.046 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:59 smithi116 bash[65980]: audit 2023-07-10T17:10:58.737826+0000 mon.a (mon.0) 153 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:00.046 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:59 smithi116 bash[65980]: cephadm 2023-07-10T17:10:58.755773+0000 mgr.y (mgr.25390) 67 : cephadm 1 Reconfiguring mgr.y (monmap changed)... 2023-07-10T17:11:00.046 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:59 smithi116 bash[65980]: audit 2023-07-10T17:10:58.756466+0000 mon.a (mon.0) 154 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:11:00.046 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:59 smithi116 bash[65980]: audit 2023-07-10T17:10:58.758308+0000 mon.a (mon.0) 155 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T17:11:00.047 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:59 smithi116 bash[65980]: audit 2023-07-10T17:10:58.760219+0000 mon.a (mon.0) 156 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:00.047 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:10:59 smithi116 bash[65980]: cephadm 2023-07-10T17:10:58.762154+0000 mgr.y (mgr.25390) 68 : cephadm 1 Reconfiguring daemon mgr.y on smithi116 2023-07-10T17:11:00.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:59 smithi192 bash[50362]: cluster 2023-07-10T17:10:58.404128+0000 mgr.y (mgr.25390) 66 : cluster 0 pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-07-10T17:11:00.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:59 smithi192 bash[50362]: audit 2023-07-10T17:10:58.728676+0000 mon.a (mon.0) 152 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:00.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:59 smithi192 bash[50362]: audit 2023-07-10T17:10:58.737826+0000 mon.a (mon.0) 153 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:00.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:59 smithi192 bash[50362]: cephadm 2023-07-10T17:10:58.755773+0000 mgr.y (mgr.25390) 67 : cephadm 1 Reconfiguring mgr.y (monmap changed)... 2023-07-10T17:11:00.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:59 smithi192 bash[50362]: audit 2023-07-10T17:10:58.756466+0000 mon.a (mon.0) 154 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:11:00.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:59 smithi192 bash[50362]: audit 2023-07-10T17:10:58.758308+0000 mon.a (mon.0) 155 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T17:11:00.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:59 smithi192 bash[50362]: audit 2023-07-10T17:10:58.760219+0000 mon.a (mon.0) 156 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:00.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:10:59 smithi192 bash[50362]: cephadm 2023-07-10T17:10:58.762154+0000 mgr.y (mgr.25390) 68 : cephadm 1 Reconfiguring daemon mgr.y on smithi116 2023-07-10T17:11:01.789 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:11:02.191 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:01 smithi116 bash[65980]: cluster 2023-07-10T17:11:00.404938+0000 mgr.y (mgr.25390) 69 : cluster 0 pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-07-10T17:11:02.191 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:01 smithi116 bash[65980]: audit 2023-07-10T17:11:00.923149+0000 mon.a (mon.0) 157 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:02.191 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:01 smithi116 bash[65980]: audit 2023-07-10T17:11:00.931292+0000 mon.a (mon.0) 158 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:02.192 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:01 smithi116 bash[65980]: cephadm 2023-07-10T17:11:00.981667+0000 mgr.y (mgr.25390) 70 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2023-07-10T17:11:02.192 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:01 smithi116 bash[65980]: audit 2023-07-10T17:11:00.982315+0000 mon.a (mon.0) 159 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:11:02.192 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:01 smithi116 bash[65980]: audit 2023-07-10T17:11:00.984205+0000 mon.a (mon.0) 160 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:11:02.192 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:01 smithi116 bash[65980]: audit 2023-07-10T17:11:00.985998+0000 mon.a (mon.0) 161 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:02.192 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:01 smithi116 bash[65980]: cephadm 2023-07-10T17:11:00.988096+0000 mgr.y (mgr.25390) 71 : cephadm 1 Reconfiguring daemon mon.c on smithi116 2023-07-10T17:11:02.192 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:01 smithi116 bash[63900]: cluster 2023-07-10T17:11:00.404938+0000 mgr.y (mgr.25390) 69 : cluster 0 pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-07-10T17:11:02.193 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:01 smithi116 bash[63900]: audit 2023-07-10T17:11:00.923149+0000 mon.a (mon.0) 157 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:02.193 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:01 smithi116 bash[63900]: audit 2023-07-10T17:11:00.931292+0000 mon.a (mon.0) 158 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:02.193 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:01 smithi116 bash[63900]: cephadm 2023-07-10T17:11:00.981667+0000 mgr.y (mgr.25390) 70 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2023-07-10T17:11:02.193 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:01 smithi116 bash[63900]: audit 2023-07-10T17:11:00.982315+0000 mon.a (mon.0) 159 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:11:02.193 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:01 smithi116 bash[63900]: audit 2023-07-10T17:11:00.984205+0000 mon.a (mon.0) 160 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:11:02.193 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:01 smithi116 bash[63900]: audit 2023-07-10T17:11:00.985998+0000 mon.a (mon.0) 161 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:02.193 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:01 smithi116 bash[63900]: cephadm 2023-07-10T17:11:00.988096+0000 mgr.y (mgr.25390) 71 : cephadm 1 Reconfiguring daemon mon.c on smithi116 2023-07-10T17:11:02.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:01 smithi192 bash[50362]: cluster 2023-07-10T17:11:00.404938+0000 mgr.y (mgr.25390) 69 : cluster 0 pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-07-10T17:11:02.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:01 smithi192 bash[50362]: audit 2023-07-10T17:11:00.923149+0000 mon.a (mon.0) 157 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:02.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:01 smithi192 bash[50362]: audit 2023-07-10T17:11:00.931292+0000 mon.a (mon.0) 158 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:02.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:01 smithi192 bash[50362]: cephadm 2023-07-10T17:11:00.981667+0000 mgr.y (mgr.25390) 70 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2023-07-10T17:11:02.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:01 smithi192 bash[50362]: audit 2023-07-10T17:11:00.982315+0000 mon.a (mon.0) 159 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:11:02.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:01 smithi192 bash[50362]: audit 2023-07-10T17:11:00.984205+0000 mon.a (mon.0) 160 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:11:02.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:01 smithi192 bash[50362]: audit 2023-07-10T17:11:00.985998+0000 mon.a (mon.0) 161 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:02.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:01 smithi192 bash[50362]: cephadm 2023-07-10T17:11:00.988096+0000 mgr.y (mgr.25390) 71 : cephadm 1 Reconfiguring daemon mon.c on smithi116 2023-07-10T17:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (4m) 48s ago 10m 10.8M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (2m) 16s ago 10m 25.6M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 error 48s ago 8m - - 2023-07-10T17:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (117s) 16s ago 14m 428M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (2m) 48s ago 16m 464M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (91s) 48s ago 16m 45.2M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (31s) 16s ago 15m 40.2M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 74a082a2e795 2023-07-10T17:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (57s) 48s ago 15m 38.1M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (4m) 48s ago 10m 9287k - 0.18.1 e5a616e4b9cf 3bd058d91353 2023-07-10T17:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (3m) 16s ago 10m 3607k - 0.18.1 e5a616e4b9cf 63551615c679 2023-07-10T17:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (14m) 48s ago 14m 55.4M 4096M 16.2.0 24ecd6d5f14c 4fa26367c830 2023-07-10T17:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (13m) 48s ago 14m 53.6M 4096M 16.2.0 24ecd6d5f14c c157509b3fd4 2023-07-10T17:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (13m) 48s ago 13m 49.4M 4096M 16.2.0 24ecd6d5f14c f1fd77b3c454 2023-07-10T17:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (13m) 48s ago 13m 54.9M 4096M 16.2.0 24ecd6d5f14c 60949d479c11 2023-07-10T17:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (12m) 16s ago 12m 55.1M 4096M 16.2.0 24ecd6d5f14c 75fc0b36aab8 2023-07-10T17:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (12m) 16s ago 12m 52.8M 4096M 16.2.0 24ecd6d5f14c d673065333dd 2023-07-10T17:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (11m) 16s ago 11m 53.7M 4096M 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (11m) 16s ago 11m 56.7M 4096M 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 error 16s ago 10m - - 2023-07-10T17:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (8m) 48s ago 8m 54.7M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (8m) 16s ago 8m 54.8M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (8m) 48s ago 8m 54.6M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (8m) 16s ago 8m 56.6M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-10T17:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 12, 2023-07-10T17:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 5 2023-07-10T17:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:11:03.364 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:11:03.364 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:11:03.366 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2023-07-10T17:11:03.366 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2023-07-10T17:11:03.366 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:11:03.366 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "5/23 daemons upgraded", 2023-07-10T17:11:03.367 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading mon daemons", 2023-07-10T17:11:03.367 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:11:03.367 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:11:03.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:02 smithi192 bash[50362]: audit 2023-07-10T17:11:01.770200+0000 mgr.y (mgr.25390) 72 : audit 0 from='client.44209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:03.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:02 smithi192 bash[50362]: audit 2023-07-10T17:11:02.148540+0000 mgr.y (mgr.25390) 73 : audit 0 from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:03.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:02 smithi116 bash[63900]: audit 2023-07-10T17:11:01.770200+0000 mgr.y (mgr.25390) 72 : audit 0 from='client.44209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:03.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:02 smithi116 bash[63900]: audit 2023-07-10T17:11:02.148540+0000 mgr.y (mgr.25390) 73 : audit 0 from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:03.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:02 smithi116 bash[65980]: audit 2023-07-10T17:11:01.770200+0000 mgr.y (mgr.25390) 72 : audit 0 from='client.44209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:03.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:02 smithi116 bash[65980]: audit 2023-07-10T17:11:02.148540+0000 mgr.y (mgr.25390) 73 : audit 0 from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:03.797 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:11:03.797 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:11:03.797 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:11:03.797 INFO:teuthology.orchestra.run.smithi116.stdout: daemon prometheus.a on smithi192 is in error state 2023-07-10T17:11:04.379 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[65980]: cluster 2023-07-10T17:11:02.406415+0000 mgr.y (mgr.25390) 74 : cluster 0 pgmap v47: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-07-10T17:11:04.379 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[65980]: audit 2023-07-10T17:11:02.512761+0000 mgr.y (mgr.25390) 75 : audit 0 from='client.44221 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:04.380 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[65980]: audit 2023-07-10T17:11:02.977757+0000 mon.a (mon.0) 162 : audit 0 from='client.34371 172.21.15.116:0/3296044479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:11:04.380 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[65980]: audit 2023-07-10T17:11:03.072653+0000 mon.a (mon.0) 163 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:04.380 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[65980]: audit 2023-07-10T17:11:03.080788+0000 mon.a (mon.0) 164 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:04.380 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[65980]: cephadm 2023-07-10T17:11:03.106643+0000 mgr.y (mgr.25390) 76 : cephadm 1 Reconfiguring osd.0 (monmap changed)... 2023-07-10T17:11:04.380 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[65980]: audit 2023-07-10T17:11:03.107128+0000 mon.a (mon.0) 165 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-10T17:11:04.380 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[65980]: audit 2023-07-10T17:11:03.108238+0000 mon.a (mon.0) 166 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:04.380 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[65980]: cephadm 2023-07-10T17:11:03.112927+0000 mgr.y (mgr.25390) 77 : cephadm 1 Reconfiguring daemon osd.0 on smithi116 2023-07-10T17:11:04.381 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[65980]: audit 2023-07-10T17:11:03.793125+0000 mon.c (mon.1) 9 : audit 0 from='client.34383 172.21.15.116:0/3210441201' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:11:04.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[63900]: cluster 2023-07-10T17:11:02.406415+0000 mgr.y (mgr.25390) 74 : cluster 0 pgmap v47: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-07-10T17:11:04.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[63900]: audit 2023-07-10T17:11:02.512761+0000 mgr.y (mgr.25390) 75 : audit 0 from='client.44221 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:04.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[63900]: audit 2023-07-10T17:11:02.977757+0000 mon.a (mon.0) 162 : audit 0 from='client.34371 172.21.15.116:0/3296044479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:11:04.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[63900]: audit 2023-07-10T17:11:03.072653+0000 mon.a (mon.0) 163 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:04.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[63900]: audit 2023-07-10T17:11:03.080788+0000 mon.a (mon.0) 164 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:04.382 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[63900]: cephadm 2023-07-10T17:11:03.106643+0000 mgr.y (mgr.25390) 76 : cephadm 1 Reconfiguring osd.0 (monmap changed)... 2023-07-10T17:11:04.382 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[63900]: audit 2023-07-10T17:11:03.107128+0000 mon.a (mon.0) 165 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-10T17:11:04.382 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[63900]: audit 2023-07-10T17:11:03.108238+0000 mon.a (mon.0) 166 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:04.382 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[63900]: cephadm 2023-07-10T17:11:03.112927+0000 mgr.y (mgr.25390) 77 : cephadm 1 Reconfiguring daemon osd.0 on smithi116 2023-07-10T17:11:04.382 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:03 smithi116 bash[63900]: audit 2023-07-10T17:11:03.793125+0000 mon.c (mon.1) 9 : audit 0 from='client.34383 172.21.15.116:0/3210441201' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:11:04.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:03 smithi192 bash[50362]: cluster 2023-07-10T17:11:02.406415+0000 mgr.y (mgr.25390) 74 : cluster 0 pgmap v47: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-07-10T17:11:04.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:03 smithi192 bash[50362]: audit 2023-07-10T17:11:02.512761+0000 mgr.y (mgr.25390) 75 : audit 0 from='client.44221 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:04.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:03 smithi192 bash[50362]: audit 2023-07-10T17:11:02.977757+0000 mon.a (mon.0) 162 : audit 0 from='client.34371 172.21.15.116:0/3296044479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:11:04.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:03 smithi192 bash[50362]: audit 2023-07-10T17:11:03.072653+0000 mon.a (mon.0) 163 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:04.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:03 smithi192 bash[50362]: audit 2023-07-10T17:11:03.080788+0000 mon.a (mon.0) 164 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:04.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:03 smithi192 bash[50362]: cephadm 2023-07-10T17:11:03.106643+0000 mgr.y (mgr.25390) 76 : cephadm 1 Reconfiguring osd.0 (monmap changed)... 2023-07-10T17:11:04.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:03 smithi192 bash[50362]: audit 2023-07-10T17:11:03.107128+0000 mon.a (mon.0) 165 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-10T17:11:04.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:03 smithi192 bash[50362]: audit 2023-07-10T17:11:03.108238+0000 mon.a (mon.0) 166 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:04.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:03 smithi192 bash[50362]: cephadm 2023-07-10T17:11:03.112927+0000 mgr.y (mgr.25390) 77 : cephadm 1 Reconfiguring daemon osd.0 on smithi116 2023-07-10T17:11:04.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:03 smithi192 bash[50362]: audit 2023-07-10T17:11:03.793125+0000 mon.c (mon.1) 9 : audit 0 from='client.34383 172.21.15.116:0/3210441201' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:11:05.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:04 smithi192 bash[50362]: audit 2023-07-10T17:11:03.359999+0000 mgr.y (mgr.25390) 78 : audit 0 from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:05.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:04 smithi116 bash[63900]: audit 2023-07-10T17:11:03.359999+0000 mgr.y (mgr.25390) 78 : audit 0 from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:05.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:04 smithi116 bash[65980]: audit 2023-07-10T17:11:03.359999+0000 mgr.y (mgr.25390) 78 : audit 0 from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:06.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:05 smithi192 bash[50362]: cluster 2023-07-10T17:11:04.407248+0000 mgr.y (mgr.25390) 79 : cluster 0 pgmap v48: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-07-10T17:11:06.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:05 smithi192 bash[50362]: audit 2023-07-10T17:11:05.285008+0000 mon.a (mon.0) 167 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:06.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:05 smithi192 bash[50362]: audit 2023-07-10T17:11:05.295090+0000 mon.a (mon.0) 168 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:06.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:05 smithi192 bash[50362]: audit 2023-07-10T17:11:05.333859+0000 mon.a (mon.0) 169 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-10T17:11:06.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:05 smithi192 bash[50362]: audit 2023-07-10T17:11:05.335729+0000 mon.a (mon.0) 170 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:06.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:05 smithi116 bash[63900]: cluster 2023-07-10T17:11:04.407248+0000 mgr.y (mgr.25390) 79 : cluster 0 pgmap v48: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-07-10T17:11:06.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:05 smithi116 bash[63900]: audit 2023-07-10T17:11:05.285008+0000 mon.a (mon.0) 167 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:06.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:05 smithi116 bash[63900]: audit 2023-07-10T17:11:05.295090+0000 mon.a (mon.0) 168 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:06.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:05 smithi116 bash[63900]: audit 2023-07-10T17:11:05.333859+0000 mon.a (mon.0) 169 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-10T17:11:06.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:05 smithi116 bash[63900]: audit 2023-07-10T17:11:05.335729+0000 mon.a (mon.0) 170 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:06.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:05 smithi116 bash[65980]: cluster 2023-07-10T17:11:04.407248+0000 mgr.y (mgr.25390) 79 : cluster 0 pgmap v48: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-07-10T17:11:06.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:05 smithi116 bash[65980]: audit 2023-07-10T17:11:05.285008+0000 mon.a (mon.0) 167 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:06.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:05 smithi116 bash[65980]: audit 2023-07-10T17:11:05.295090+0000 mon.a (mon.0) 168 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:06.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:05 smithi116 bash[65980]: audit 2023-07-10T17:11:05.333859+0000 mon.a (mon.0) 169 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-10T17:11:06.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:05 smithi116 bash[65980]: audit 2023-07-10T17:11:05.335729+0000 mon.a (mon.0) 170 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:07.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:06 smithi116 bash[63900]: cephadm 2023-07-10T17:11:05.333180+0000 mgr.y (mgr.25390) 80 : cephadm 1 Reconfiguring osd.1 (monmap changed)... 2023-07-10T17:11:07.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:06 smithi116 bash[63900]: cephadm 2023-07-10T17:11:05.341447+0000 mgr.y (mgr.25390) 81 : cephadm 1 Reconfiguring daemon osd.1 on smithi116 2023-07-10T17:11:07.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:06 smithi116 bash[65980]: cephadm 2023-07-10T17:11:05.333180+0000 mgr.y (mgr.25390) 80 : cephadm 1 Reconfiguring osd.1 (monmap changed)... 2023-07-10T17:11:07.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:06 smithi116 bash[65980]: cephadm 2023-07-10T17:11:05.341447+0000 mgr.y (mgr.25390) 81 : cephadm 1 Reconfiguring daemon osd.1 on smithi116 2023-07-10T17:11:07.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:06 smithi192 bash[50362]: cephadm 2023-07-10T17:11:05.333180+0000 mgr.y (mgr.25390) 80 : cephadm 1 Reconfiguring osd.1 (monmap changed)... 2023-07-10T17:11:07.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:06 smithi192 bash[50362]: cephadm 2023-07-10T17:11:05.341447+0000 mgr.y (mgr.25390) 81 : cephadm 1 Reconfiguring daemon osd.1 on smithi116 2023-07-10T17:11:08.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:07 smithi192 bash[50362]: cluster 2023-07-10T17:11:06.407949+0000 mgr.y (mgr.25390) 82 : cluster 0 pgmap v49: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-07-10T17:11:08.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:07 smithi116 bash[63900]: cluster 2023-07-10T17:11:06.407949+0000 mgr.y (mgr.25390) 82 : cluster 0 pgmap v49: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-07-10T17:11:08.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:07 smithi116 bash[65980]: cluster 2023-07-10T17:11:06.407949+0000 mgr.y (mgr.25390) 82 : cluster 0 pgmap v49: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-07-10T17:11:10.312 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:09 smithi116 bash[65980]: cluster 2023-07-10T17:11:08.409743+0000 mgr.y (mgr.25390) 83 : cluster 0 pgmap v50: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:10.313 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:09 smithi116 bash[65980]: audit 2023-07-10T17:11:08.946726+0000 mon.a (mon.0) 171 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:10.313 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:09 smithi116 bash[65980]: audit 2023-07-10T17:11:08.957488+0000 mon.a (mon.0) 172 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:10.313 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:09 smithi116 bash[65980]: cephadm 2023-07-10T17:11:08.963945+0000 mgr.y (mgr.25390) 84 : cephadm 1 Reconfiguring osd.2 (monmap changed)... 2023-07-10T17:11:10.313 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:09 smithi116 bash[65980]: audit 2023-07-10T17:11:08.964433+0000 mon.a (mon.0) 173 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-10T17:11:10.313 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:09 smithi116 bash[65980]: audit 2023-07-10T17:11:08.970891+0000 mon.a (mon.0) 174 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:10.313 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:09 smithi116 bash[65980]: cephadm 2023-07-10T17:11:08.975309+0000 mgr.y (mgr.25390) 85 : cephadm 1 Reconfiguring daemon osd.2 on smithi116 2023-07-10T17:11:10.313 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:09 smithi116 bash[65980]: audit 2023-07-10T17:11:09.926018+0000 mon.c (mon.1) 10 : audit 0 from='client.34401 172.21.15.116:0/283784903' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:11:10.314 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:09 smithi116 bash[63900]: cluster 2023-07-10T17:11:08.409743+0000 mgr.y (mgr.25390) 83 : cluster 0 pgmap v50: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:10.314 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:09 smithi116 bash[63900]: audit 2023-07-10T17:11:08.946726+0000 mon.a (mon.0) 171 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:10.314 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:09 smithi116 bash[63900]: audit 2023-07-10T17:11:08.957488+0000 mon.a (mon.0) 172 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:10.314 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:09 smithi116 bash[63900]: cephadm 2023-07-10T17:11:08.963945+0000 mgr.y (mgr.25390) 84 : cephadm 1 Reconfiguring osd.2 (monmap changed)... 2023-07-10T17:11:10.314 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:09 smithi116 bash[63900]: audit 2023-07-10T17:11:08.964433+0000 mon.a (mon.0) 173 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-10T17:11:10.314 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:09 smithi116 bash[63900]: audit 2023-07-10T17:11:08.970891+0000 mon.a (mon.0) 174 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:10.315 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:09 smithi116 bash[63900]: cephadm 2023-07-10T17:11:08.975309+0000 mgr.y (mgr.25390) 85 : cephadm 1 Reconfiguring daemon osd.2 on smithi116 2023-07-10T17:11:10.315 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:09 smithi116 bash[63900]: audit 2023-07-10T17:11:09.926018+0000 mon.c (mon.1) 10 : audit 0 from='client.34401 172.21.15.116:0/283784903' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:11:10.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:09 smithi192 bash[50362]: cluster 2023-07-10T17:11:08.409743+0000 mgr.y (mgr.25390) 83 : cluster 0 pgmap v50: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:10.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:09 smithi192 bash[50362]: audit 2023-07-10T17:11:08.946726+0000 mon.a (mon.0) 171 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:10.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:09 smithi192 bash[50362]: audit 2023-07-10T17:11:08.957488+0000 mon.a (mon.0) 172 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:10.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:09 smithi192 bash[50362]: cephadm 2023-07-10T17:11:08.963945+0000 mgr.y (mgr.25390) 84 : cephadm 1 Reconfiguring osd.2 (monmap changed)... 2023-07-10T17:11:10.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:09 smithi192 bash[50362]: audit 2023-07-10T17:11:08.964433+0000 mon.a (mon.0) 173 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-10T17:11:10.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:09 smithi192 bash[50362]: audit 2023-07-10T17:11:08.970891+0000 mon.a (mon.0) 174 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:10.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:09 smithi192 bash[50362]: cephadm 2023-07-10T17:11:08.975309+0000 mgr.y (mgr.25390) 85 : cephadm 1 Reconfiguring daemon osd.2 on smithi116 2023-07-10T17:11:10.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:09 smithi192 bash[50362]: audit 2023-07-10T17:11:09.926018+0000 mon.c (mon.1) 10 : audit 0 from='client.34401 172.21.15.116:0/283784903' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2023-07-10T17:11:11.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:10 smithi192 bash[50362]: audit 2023-07-10T17:11:10.303743+0000 mon.c (mon.1) 11 : audit 1 from='client.44263 172.21.15.116:0/764199035' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]: access denied 2023-07-10T17:11:11.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:10 smithi116 bash[63900]: audit 2023-07-10T17:11:10.303743+0000 mon.c (mon.1) 11 : audit 1 from='client.44263 172.21.15.116:0/764199035' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]: access denied 2023-07-10T17:11:11.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:10 smithi116 bash[65980]: audit 2023-07-10T17:11:10.303743+0000 mon.c (mon.1) 11 : audit 1 from='client.44263 172.21.15.116:0/764199035' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]: access denied 2023-07-10T17:11:12.348 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:11 smithi116 bash[65980]: cluster 2023-07-10T17:11:10.410551+0000 mgr.y (mgr.25390) 86 : cluster 0 pgmap v51: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:12.348 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:11 smithi116 bash[63900]: cluster 2023-07-10T17:11:10.410551+0000 mgr.y (mgr.25390) 86 : cluster 0 pgmap v51: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:12.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:11 smithi192 bash[50362]: cluster 2023-07-10T17:11:10.410551+0000 mgr.y (mgr.25390) 86 : cluster 0 pgmap v51: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:14.286 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:13 smithi116 bash[65980]: cluster 2023-07-10T17:11:12.412344+0000 mgr.y (mgr.25390) 87 : cluster 0 pgmap v52: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:14.286 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:13 smithi116 bash[65980]: audit 2023-07-10T17:11:12.914129+0000 mon.a (mon.0) 175 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:14.287 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:13 smithi116 bash[65980]: audit 2023-07-10T17:11:12.921065+0000 mon.a (mon.0) 176 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:14.287 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:13 smithi116 bash[65980]: cephadm 2023-07-10T17:11:12.947870+0000 mgr.y (mgr.25390) 88 : cephadm 1 Reconfiguring osd.3 (monmap changed)... 2023-07-10T17:11:14.287 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:13 smithi116 bash[65980]: audit 2023-07-10T17:11:12.948587+0000 mon.a (mon.0) 177 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-10T17:11:14.287 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:13 smithi116 bash[65980]: audit 2023-07-10T17:11:12.950692+0000 mon.a (mon.0) 178 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:14.287 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:13 smithi116 bash[65980]: cephadm 2023-07-10T17:11:12.955999+0000 mgr.y (mgr.25390) 89 : cephadm 1 Reconfiguring daemon osd.3 on smithi116 2023-07-10T17:11:14.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:13 smithi116 bash[63900]: cluster 2023-07-10T17:11:12.412344+0000 mgr.y (mgr.25390) 87 : cluster 0 pgmap v52: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:14.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:13 smithi116 bash[63900]: audit 2023-07-10T17:11:12.914129+0000 mon.a (mon.0) 175 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:14.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:13 smithi116 bash[63900]: audit 2023-07-10T17:11:12.921065+0000 mon.a (mon.0) 176 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:14.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:13 smithi116 bash[63900]: cephadm 2023-07-10T17:11:12.947870+0000 mgr.y (mgr.25390) 88 : cephadm 1 Reconfiguring osd.3 (monmap changed)... 2023-07-10T17:11:14.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:13 smithi116 bash[63900]: audit 2023-07-10T17:11:12.948587+0000 mon.a (mon.0) 177 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-10T17:11:14.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:13 smithi116 bash[63900]: audit 2023-07-10T17:11:12.950692+0000 mon.a (mon.0) 178 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:14.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:13 smithi116 bash[63900]: cephadm 2023-07-10T17:11:12.955999+0000 mgr.y (mgr.25390) 89 : cephadm 1 Reconfiguring daemon osd.3 on smithi116 2023-07-10T17:11:14.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:13 smithi192 bash[50362]: cluster 2023-07-10T17:11:12.412344+0000 mgr.y (mgr.25390) 87 : cluster 0 pgmap v52: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:14.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:13 smithi192 bash[50362]: audit 2023-07-10T17:11:12.914129+0000 mon.a (mon.0) 175 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:14.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:13 smithi192 bash[50362]: audit 2023-07-10T17:11:12.921065+0000 mon.a (mon.0) 176 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:14.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:13 smithi192 bash[50362]: cephadm 2023-07-10T17:11:12.947870+0000 mgr.y (mgr.25390) 88 : cephadm 1 Reconfiguring osd.3 (monmap changed)... 2023-07-10T17:11:14.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:13 smithi192 bash[50362]: audit 2023-07-10T17:11:12.948587+0000 mon.a (mon.0) 177 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-10T17:11:14.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:13 smithi192 bash[50362]: audit 2023-07-10T17:11:12.950692+0000 mon.a (mon.0) 178 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:14.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:13 smithi192 bash[50362]: cephadm 2023-07-10T17:11:12.955999+0000 mgr.y (mgr.25390) 89 : cephadm 1 Reconfiguring daemon osd.3 on smithi116 2023-07-10T17:11:16.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:15 smithi192 bash[50362]: cluster 2023-07-10T17:11:14.413268+0000 mgr.y (mgr.25390) 90 : cluster 0 pgmap v53: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:16.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:15 smithi192 bash[50362]: audit 2023-07-10T17:11:15.236334+0000 mon.a (mon.0) 179 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:16.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:15 smithi192 bash[50362]: audit 2023-07-10T17:11:15.244141+0000 mon.a (mon.0) 180 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:16.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:15 smithi192 bash[50362]: audit 2023-07-10T17:11:15.277715+0000 mon.a (mon.0) 181 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi116.stnxgn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:11:16.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:15 smithi192 bash[50362]: audit 2023-07-10T17:11:15.281800+0000 mon.a (mon.0) 182 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:16.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:15 smithi116 bash[63900]: cluster 2023-07-10T17:11:14.413268+0000 mgr.y (mgr.25390) 90 : cluster 0 pgmap v53: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:16.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:15 smithi116 bash[63900]: audit 2023-07-10T17:11:15.236334+0000 mon.a (mon.0) 179 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:16.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:15 smithi116 bash[63900]: audit 2023-07-10T17:11:15.244141+0000 mon.a (mon.0) 180 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:16.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:15 smithi116 bash[63900]: audit 2023-07-10T17:11:15.277715+0000 mon.a (mon.0) 181 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi116.stnxgn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:11:16.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:15 smithi116 bash[63900]: audit 2023-07-10T17:11:15.281800+0000 mon.a (mon.0) 182 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:16.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:15 smithi116 bash[65980]: cluster 2023-07-10T17:11:14.413268+0000 mgr.y (mgr.25390) 90 : cluster 0 pgmap v53: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:16.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:15 smithi116 bash[65980]: audit 2023-07-10T17:11:15.236334+0000 mon.a (mon.0) 179 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:16.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:15 smithi116 bash[65980]: audit 2023-07-10T17:11:15.244141+0000 mon.a (mon.0) 180 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:16.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:15 smithi116 bash[65980]: audit 2023-07-10T17:11:15.277715+0000 mon.a (mon.0) 181 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi116.stnxgn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:11:16.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:15 smithi116 bash[65980]: audit 2023-07-10T17:11:15.281800+0000 mon.a (mon.0) 182 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:17.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:16 smithi116 bash[63900]: cephadm 2023-07-10T17:11:15.276894+0000 mgr.y (mgr.25390) 91 : cephadm 1 Reconfiguring rgw.foo.smithi116.stnxgn (monmap changed)... 2023-07-10T17:11:17.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:16 smithi116 bash[63900]: cephadm 2023-07-10T17:11:15.283844+0000 mgr.y (mgr.25390) 92 : cephadm 1 Reconfiguring daemon rgw.foo.smithi116.stnxgn on smithi116 2023-07-10T17:11:17.072 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:16 smithi116 bash[65980]: cephadm 2023-07-10T17:11:15.276894+0000 mgr.y (mgr.25390) 91 : cephadm 1 Reconfiguring rgw.foo.smithi116.stnxgn (monmap changed)... 2023-07-10T17:11:17.073 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:16 smithi116 bash[65980]: cephadm 2023-07-10T17:11:15.283844+0000 mgr.y (mgr.25390) 92 : cephadm 1 Reconfiguring daemon rgw.foo.smithi116.stnxgn on smithi116 2023-07-10T17:11:17.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:16 smithi192 bash[50362]: cephadm 2023-07-10T17:11:15.276894+0000 mgr.y (mgr.25390) 91 : cephadm 1 Reconfiguring rgw.foo.smithi116.stnxgn (monmap changed)... 2023-07-10T17:11:17.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:16 smithi192 bash[50362]: cephadm 2023-07-10T17:11:15.283844+0000 mgr.y (mgr.25390) 92 : cephadm 1 Reconfiguring daemon rgw.foo.smithi116.stnxgn on smithi116 2023-07-10T17:11:18.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:17 smithi192 bash[50362]: cluster 2023-07-10T17:11:16.414191+0000 mgr.y (mgr.25390) 93 : cluster 0 pgmap v54: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:18.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:17 smithi192 bash[50362]: audit 2023-07-10T17:11:17.701852+0000 mon.a (mon.0) 183 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:18.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:17 smithi192 bash[50362]: audit 2023-07-10T17:11:17.709488+0000 mon.a (mon.0) 184 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:18.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:17 smithi192 bash[50362]: audit 2023-07-10T17:11:17.754365+0000 mon.a (mon.0) 185 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi116.vpoihf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:11:18.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:17 smithi192 bash[50362]: audit 2023-07-10T17:11:17.758306+0000 mon.a (mon.0) 186 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:18.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:17 smithi116 bash[63900]: cluster 2023-07-10T17:11:16.414191+0000 mgr.y (mgr.25390) 93 : cluster 0 pgmap v54: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:18.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:17 smithi116 bash[63900]: audit 2023-07-10T17:11:17.701852+0000 mon.a (mon.0) 183 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:18.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:17 smithi116 bash[63900]: audit 2023-07-10T17:11:17.709488+0000 mon.a (mon.0) 184 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:18.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:17 smithi116 bash[63900]: audit 2023-07-10T17:11:17.754365+0000 mon.a (mon.0) 185 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi116.vpoihf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:11:18.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:17 smithi116 bash[63900]: audit 2023-07-10T17:11:17.758306+0000 mon.a (mon.0) 186 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:18.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:17 smithi116 bash[65980]: cluster 2023-07-10T17:11:16.414191+0000 mgr.y (mgr.25390) 93 : cluster 0 pgmap v54: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:18.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:17 smithi116 bash[65980]: audit 2023-07-10T17:11:17.701852+0000 mon.a (mon.0) 183 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:18.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:17 smithi116 bash[65980]: audit 2023-07-10T17:11:17.709488+0000 mon.a (mon.0) 184 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:18.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:17 smithi116 bash[65980]: audit 2023-07-10T17:11:17.754365+0000 mon.a (mon.0) 185 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi116.vpoihf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:11:18.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:17 smithi116 bash[65980]: audit 2023-07-10T17:11:17.758306+0000 mon.a (mon.0) 186 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:19.237 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:18 smithi116 bash[65980]: cephadm 2023-07-10T17:11:17.753592+0000 mgr.y (mgr.25390) 94 : cephadm 1 Reconfiguring rgw.smpl.smithi116.vpoihf (monmap changed)... 2023-07-10T17:11:19.238 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:18 smithi116 bash[65980]: cephadm 2023-07-10T17:11:17.760390+0000 mgr.y (mgr.25390) 95 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi116.vpoihf on smithi116 2023-07-10T17:11:19.238 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:18 smithi116 bash[63900]: cephadm 2023-07-10T17:11:17.753592+0000 mgr.y (mgr.25390) 94 : cephadm 1 Reconfiguring rgw.smpl.smithi116.vpoihf (monmap changed)... 2023-07-10T17:11:19.239 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:18 smithi116 bash[63900]: cephadm 2023-07-10T17:11:17.760390+0000 mgr.y (mgr.25390) 95 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi116.vpoihf on smithi116 2023-07-10T17:11:19.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:18 smithi192 bash[50362]: cephadm 2023-07-10T17:11:17.753592+0000 mgr.y (mgr.25390) 94 : cephadm 1 Reconfiguring rgw.smpl.smithi116.vpoihf (monmap changed)... 2023-07-10T17:11:19.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:18 smithi192 bash[50362]: cephadm 2023-07-10T17:11:17.760390+0000 mgr.y (mgr.25390) 95 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi116.vpoihf on smithi116 2023-07-10T17:11:20.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:19 smithi192 bash[50362]: cluster 2023-07-10T17:11:18.415943+0000 mgr.y (mgr.25390) 96 : cluster 0 pgmap v55: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:20.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:19 smithi116 bash[63900]: cluster 2023-07-10T17:11:18.415943+0000 mgr.y (mgr.25390) 96 : cluster 0 pgmap v55: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:20.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:19 smithi116 bash[65980]: cluster 2023-07-10T17:11:18.415943+0000 mgr.y (mgr.25390) 96 : cluster 0 pgmap v55: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:21.208 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:20 smithi192 bash[50362]: audit 2023-07-10T17:11:19.950559+0000 mon.a (mon.0) 187 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:21.208 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:20 smithi192 bash[50362]: audit 2023-07-10T17:11:19.961001+0000 mon.a (mon.0) 188 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:21.208 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:20 smithi192 bash[50362]: cephadm 2023-07-10T17:11:19.979698+0000 mgr.y (mgr.25390) 97 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2023-07-10T17:11:21.209 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:20 smithi192 bash[50362]: audit 2023-07-10T17:11:19.980251+0000 mon.a (mon.0) 189 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:11:21.209 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:20 smithi192 bash[50362]: audit 2023-07-10T17:11:19.981873+0000 mon.a (mon.0) 190 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:11:21.209 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:20 smithi192 bash[50362]: audit 2023-07-10T17:11:19.983300+0000 mon.a (mon.0) 191 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:21.209 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:20 smithi192 bash[50362]: cephadm 2023-07-10T17:11:19.984979+0000 mgr.y (mgr.25390) 98 : cephadm 1 Reconfiguring daemon mon.b on smithi192 2023-07-10T17:11:21.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:20 smithi116 bash[63900]: audit 2023-07-10T17:11:19.950559+0000 mon.a (mon.0) 187 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:21.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:20 smithi116 bash[63900]: audit 2023-07-10T17:11:19.961001+0000 mon.a (mon.0) 188 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:21.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:20 smithi116 bash[63900]: cephadm 2023-07-10T17:11:19.979698+0000 mgr.y (mgr.25390) 97 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2023-07-10T17:11:21.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:20 smithi116 bash[63900]: audit 2023-07-10T17:11:19.980251+0000 mon.a (mon.0) 189 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:11:21.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:20 smithi116 bash[63900]: audit 2023-07-10T17:11:19.981873+0000 mon.a (mon.0) 190 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:11:21.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:20 smithi116 bash[63900]: audit 2023-07-10T17:11:19.983300+0000 mon.a (mon.0) 191 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:21.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:20 smithi116 bash[63900]: cephadm 2023-07-10T17:11:19.984979+0000 mgr.y (mgr.25390) 98 : cephadm 1 Reconfiguring daemon mon.b on smithi192 2023-07-10T17:11:21.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:20 smithi116 bash[65980]: audit 2023-07-10T17:11:19.950559+0000 mon.a (mon.0) 187 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:21.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:20 smithi116 bash[65980]: audit 2023-07-10T17:11:19.961001+0000 mon.a (mon.0) 188 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:21.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:20 smithi116 bash[65980]: cephadm 2023-07-10T17:11:19.979698+0000 mgr.y (mgr.25390) 97 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2023-07-10T17:11:21.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:20 smithi116 bash[65980]: audit 2023-07-10T17:11:19.980251+0000 mon.a (mon.0) 189 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-10T17:11:21.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:20 smithi116 bash[65980]: audit 2023-07-10T17:11:19.981873+0000 mon.a (mon.0) 190 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-10T17:11:21.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:20 smithi116 bash[65980]: audit 2023-07-10T17:11:19.983300+0000 mon.a (mon.0) 191 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:21.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:20 smithi116 bash[65980]: cephadm 2023-07-10T17:11:19.984979+0000 mgr.y (mgr.25390) 98 : cephadm 1 Reconfiguring daemon mon.b on smithi192 2023-07-10T17:11:22.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:21 smithi192 bash[50362]: cluster 2023-07-10T17:11:20.416698+0000 mgr.y (mgr.25390) 99 : cluster 0 pgmap v56: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:22.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:21 smithi116 bash[63900]: cluster 2023-07-10T17:11:20.416698+0000 mgr.y (mgr.25390) 99 : cluster 0 pgmap v56: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:22.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:21 smithi116 bash[65980]: cluster 2023-07-10T17:11:20.416698+0000 mgr.y (mgr.25390) 99 : cluster 0 pgmap v56: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:23.356 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:23 smithi192 bash[50362]: audit 2023-07-10T17:11:22.072062+0000 mon.a (mon.0) 192 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:23.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:23 smithi192 bash[50362]: audit 2023-07-10T17:11:22.084243+0000 mon.a (mon.0) 193 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:23.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:23 smithi192 bash[50362]: cephadm 2023-07-10T17:11:22.105084+0000 mgr.y (mgr.25390) 100 : cephadm 1 Reconfiguring mgr.x (monmap changed)... 2023-07-10T17:11:23.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:23 smithi192 bash[50362]: audit 2023-07-10T17:11:22.105738+0000 mon.a (mon.0) 194 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:11:23.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:23 smithi192 bash[50362]: audit 2023-07-10T17:11:22.107730+0000 mon.a (mon.0) 195 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T17:11:23.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:23 smithi192 bash[50362]: audit 2023-07-10T17:11:22.109422+0000 mon.a (mon.0) 196 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:23.358 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:23 smithi192 bash[50362]: cephadm 2023-07-10T17:11:22.111201+0000 mgr.y (mgr.25390) 101 : cephadm 1 Reconfiguring daemon mgr.x on smithi192 2023-07-10T17:11:23.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:23 smithi116 bash[63900]: audit 2023-07-10T17:11:22.072062+0000 mon.a (mon.0) 192 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:23.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:23 smithi116 bash[63900]: audit 2023-07-10T17:11:22.084243+0000 mon.a (mon.0) 193 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:23.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:23 smithi116 bash[63900]: cephadm 2023-07-10T17:11:22.105084+0000 mgr.y (mgr.25390) 100 : cephadm 1 Reconfiguring mgr.x (monmap changed)... 2023-07-10T17:11:23.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:23 smithi116 bash[63900]: audit 2023-07-10T17:11:22.105738+0000 mon.a (mon.0) 194 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:11:23.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:23 smithi116 bash[63900]: audit 2023-07-10T17:11:22.107730+0000 mon.a (mon.0) 195 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T17:11:23.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:23 smithi116 bash[63900]: audit 2023-07-10T17:11:22.109422+0000 mon.a (mon.0) 196 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:23.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:23 smithi116 bash[63900]: cephadm 2023-07-10T17:11:22.111201+0000 mgr.y (mgr.25390) 101 : cephadm 1 Reconfiguring daemon mgr.x on smithi192 2023-07-10T17:11:23.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:23 smithi116 bash[65980]: audit 2023-07-10T17:11:22.072062+0000 mon.a (mon.0) 192 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:23.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:23 smithi116 bash[65980]: audit 2023-07-10T17:11:22.084243+0000 mon.a (mon.0) 193 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:23.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:23 smithi116 bash[65980]: cephadm 2023-07-10T17:11:22.105084+0000 mgr.y (mgr.25390) 100 : cephadm 1 Reconfiguring mgr.x (monmap changed)... 2023-07-10T17:11:23.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:23 smithi116 bash[65980]: audit 2023-07-10T17:11:22.105738+0000 mon.a (mon.0) 194 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-10T17:11:23.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:23 smithi116 bash[65980]: audit 2023-07-10T17:11:22.107730+0000 mon.a (mon.0) 195 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-10T17:11:23.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:23 smithi116 bash[65980]: audit 2023-07-10T17:11:22.109422+0000 mon.a (mon.0) 196 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:23.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:23 smithi116 bash[65980]: cephadm 2023-07-10T17:11:22.111201+0000 mgr.y (mgr.25390) 101 : cephadm 1 Reconfiguring daemon mgr.x on smithi192 2023-07-10T17:11:24.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:24 smithi192 bash[50362]: cluster 2023-07-10T17:11:22.418313+0000 mgr.y (mgr.25390) 102 : cluster 0 pgmap v57: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:24.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:24 smithi116 bash[63900]: cluster 2023-07-10T17:11:22.418313+0000 mgr.y (mgr.25390) 102 : cluster 0 pgmap v57: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:24.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:24 smithi116 bash[65980]: cluster 2023-07-10T17:11:22.418313+0000 mgr.y (mgr.25390) 102 : cluster 0 pgmap v57: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-07-10T17:11:25.524 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:25 smithi192 bash[50362]: audit 2023-07-10T17:11:24.206230+0000 mon.a (mon.0) 197 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:25.525 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:25 smithi192 bash[50362]: audit 2023-07-10T17:11:24.220174+0000 mon.a (mon.0) 198 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:25.525 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:25 smithi192 bash[50362]: audit 2023-07-10T17:11:24.232660+0000 mon.a (mon.0) 199 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-10T17:11:25.525 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:25 smithi192 bash[50362]: audit 2023-07-10T17:11:24.234291+0000 mon.a (mon.0) 200 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:25.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:25 smithi116 bash[65980]: audit 2023-07-10T17:11:24.206230+0000 mon.a (mon.0) 197 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:25.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:25 smithi116 bash[65980]: audit 2023-07-10T17:11:24.220174+0000 mon.a (mon.0) 198 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:25.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:25 smithi116 bash[65980]: audit 2023-07-10T17:11:24.232660+0000 mon.a (mon.0) 199 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-10T17:11:25.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:25 smithi116 bash[65980]: audit 2023-07-10T17:11:24.234291+0000 mon.a (mon.0) 200 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:25.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:25 smithi116 bash[63900]: audit 2023-07-10T17:11:24.206230+0000 mon.a (mon.0) 197 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:25.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:25 smithi116 bash[63900]: audit 2023-07-10T17:11:24.220174+0000 mon.a (mon.0) 198 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:25.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:25 smithi116 bash[63900]: audit 2023-07-10T17:11:24.232660+0000 mon.a (mon.0) 199 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-10T17:11:25.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:25 smithi116 bash[63900]: audit 2023-07-10T17:11:24.234291+0000 mon.a (mon.0) 200 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:26.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:26 smithi192 bash[50362]: cephadm 2023-07-10T17:11:24.232038+0000 mgr.y (mgr.25390) 103 : cephadm 1 Reconfiguring osd.4 (monmap changed)... 2023-07-10T17:11:26.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:26 smithi192 bash[50362]: cephadm 2023-07-10T17:11:24.239364+0000 mgr.y (mgr.25390) 104 : cephadm 1 Reconfiguring daemon osd.4 on smithi192 2023-07-10T17:11:26.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:26 smithi192 bash[50362]: cluster 2023-07-10T17:11:24.419081+0000 mgr.y (mgr.25390) 105 : cluster 0 pgmap v58: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:26.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:26 smithi116 bash[63900]: cephadm 2023-07-10T17:11:24.232038+0000 mgr.y (mgr.25390) 103 : cephadm 1 Reconfiguring osd.4 (monmap changed)... 2023-07-10T17:11:26.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:26 smithi116 bash[63900]: cephadm 2023-07-10T17:11:24.239364+0000 mgr.y (mgr.25390) 104 : cephadm 1 Reconfiguring daemon osd.4 on smithi192 2023-07-10T17:11:26.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:26 smithi116 bash[63900]: cluster 2023-07-10T17:11:24.419081+0000 mgr.y (mgr.25390) 105 : cluster 0 pgmap v58: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:26.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:26 smithi116 bash[65980]: cephadm 2023-07-10T17:11:24.232038+0000 mgr.y (mgr.25390) 103 : cephadm 1 Reconfiguring osd.4 (monmap changed)... 2023-07-10T17:11:26.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:26 smithi116 bash[65980]: cephadm 2023-07-10T17:11:24.239364+0000 mgr.y (mgr.25390) 104 : cephadm 1 Reconfiguring daemon osd.4 on smithi192 2023-07-10T17:11:26.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:26 smithi116 bash[65980]: cluster 2023-07-10T17:11:24.419081+0000 mgr.y (mgr.25390) 105 : cluster 0 pgmap v58: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:27.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:27 smithi192 bash[50362]: audit 2023-07-10T17:11:26.406672+0000 mon.a (mon.0) 201 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:27.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:27 smithi192 bash[50362]: cluster 2023-07-10T17:11:26.419817+0000 mgr.y (mgr.25390) 106 : cluster 0 pgmap v59: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:27.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:27 smithi192 bash[50362]: audit 2023-07-10T17:11:26.420439+0000 mon.a (mon.0) 202 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:27.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:27 smithi192 bash[50362]: cephadm 2023-07-10T17:11:26.458621+0000 mgr.y (mgr.25390) 107 : cephadm 1 Reconfiguring osd.5 (monmap changed)... 2023-07-10T17:11:27.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:27 smithi192 bash[50362]: audit 2023-07-10T17:11:26.459575+0000 mon.a (mon.0) 203 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-10T17:11:27.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:27 smithi192 bash[50362]: audit 2023-07-10T17:11:26.461520+0000 mon.a (mon.0) 204 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:27.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:27 smithi192 bash[50362]: cephadm 2023-07-10T17:11:26.466903+0000 mgr.y (mgr.25390) 108 : cephadm 1 Reconfiguring daemon osd.5 on smithi192 2023-07-10T17:11:27.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:27 smithi116 bash[63900]: audit 2023-07-10T17:11:26.406672+0000 mon.a (mon.0) 201 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:27.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:27 smithi116 bash[63900]: cluster 2023-07-10T17:11:26.419817+0000 mgr.y (mgr.25390) 106 : cluster 0 pgmap v59: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:27.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:27 smithi116 bash[63900]: audit 2023-07-10T17:11:26.420439+0000 mon.a (mon.0) 202 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:27.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:27 smithi116 bash[63900]: cephadm 2023-07-10T17:11:26.458621+0000 mgr.y (mgr.25390) 107 : cephadm 1 Reconfiguring osd.5 (monmap changed)... 2023-07-10T17:11:27.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:27 smithi116 bash[63900]: audit 2023-07-10T17:11:26.459575+0000 mon.a (mon.0) 203 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-10T17:11:27.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:27 smithi116 bash[63900]: audit 2023-07-10T17:11:26.461520+0000 mon.a (mon.0) 204 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:27.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:27 smithi116 bash[63900]: cephadm 2023-07-10T17:11:26.466903+0000 mgr.y (mgr.25390) 108 : cephadm 1 Reconfiguring daemon osd.5 on smithi192 2023-07-10T17:11:27.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:27 smithi116 bash[65980]: audit 2023-07-10T17:11:26.406672+0000 mon.a (mon.0) 201 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:27.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:27 smithi116 bash[65980]: cluster 2023-07-10T17:11:26.419817+0000 mgr.y (mgr.25390) 106 : cluster 0 pgmap v59: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:27.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:27 smithi116 bash[65980]: audit 2023-07-10T17:11:26.420439+0000 mon.a (mon.0) 202 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:27.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:27 smithi116 bash[65980]: cephadm 2023-07-10T17:11:26.458621+0000 mgr.y (mgr.25390) 107 : cephadm 1 Reconfiguring osd.5 (monmap changed)... 2023-07-10T17:11:27.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:27 smithi116 bash[65980]: audit 2023-07-10T17:11:26.459575+0000 mon.a (mon.0) 203 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-10T17:11:27.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:27 smithi116 bash[65980]: audit 2023-07-10T17:11:26.461520+0000 mon.a (mon.0) 204 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:27.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:27 smithi116 bash[65980]: cephadm 2023-07-10T17:11:26.466903+0000 mgr.y (mgr.25390) 108 : cephadm 1 Reconfiguring daemon osd.5 on smithi192 2023-07-10T17:11:30.187 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:29 smithi192 bash[50362]: cluster 2023-07-10T17:11:28.421497+0000 mgr.y (mgr.25390) 109 : cluster 0 pgmap v60: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:30.187 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:29 smithi192 bash[50362]: audit 2023-07-10T17:11:28.894427+0000 mon.a (mon.0) 205 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:30.187 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:29 smithi192 bash[50362]: audit 2023-07-10T17:11:28.906148+0000 mon.a (mon.0) 206 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:30.188 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:29 smithi192 bash[50362]: cephadm 2023-07-10T17:11:28.937241+0000 mgr.y (mgr.25390) 110 : cephadm 1 Reconfiguring osd.6 (monmap changed)... 2023-07-10T17:11:30.188 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:29 smithi192 bash[50362]: audit 2023-07-10T17:11:28.938004+0000 mon.a (mon.0) 207 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-10T17:11:30.188 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:29 smithi192 bash[50362]: audit 2023-07-10T17:11:28.940266+0000 mon.a (mon.0) 208 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:30.189 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:29 smithi192 bash[50362]: cephadm 2023-07-10T17:11:28.945890+0000 mgr.y (mgr.25390) 111 : cephadm 1 Reconfiguring daemon osd.6 on smithi192 2023-07-10T17:11:30.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:29 smithi116 bash[63900]: cluster 2023-07-10T17:11:28.421497+0000 mgr.y (mgr.25390) 109 : cluster 0 pgmap v60: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:30.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:29 smithi116 bash[63900]: audit 2023-07-10T17:11:28.894427+0000 mon.a (mon.0) 205 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:30.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:29 smithi116 bash[63900]: audit 2023-07-10T17:11:28.906148+0000 mon.a (mon.0) 206 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:30.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:29 smithi116 bash[63900]: cephadm 2023-07-10T17:11:28.937241+0000 mgr.y (mgr.25390) 110 : cephadm 1 Reconfiguring osd.6 (monmap changed)... 2023-07-10T17:11:30.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:29 smithi116 bash[63900]: audit 2023-07-10T17:11:28.938004+0000 mon.a (mon.0) 207 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-10T17:11:30.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:29 smithi116 bash[63900]: audit 2023-07-10T17:11:28.940266+0000 mon.a (mon.0) 208 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:30.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:29 smithi116 bash[63900]: cephadm 2023-07-10T17:11:28.945890+0000 mgr.y (mgr.25390) 111 : cephadm 1 Reconfiguring daemon osd.6 on smithi192 2023-07-10T17:11:30.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:29 smithi116 bash[65980]: cluster 2023-07-10T17:11:28.421497+0000 mgr.y (mgr.25390) 109 : cluster 0 pgmap v60: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:30.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:29 smithi116 bash[65980]: audit 2023-07-10T17:11:28.894427+0000 mon.a (mon.0) 205 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:30.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:29 smithi116 bash[65980]: audit 2023-07-10T17:11:28.906148+0000 mon.a (mon.0) 206 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:30.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:29 smithi116 bash[65980]: cephadm 2023-07-10T17:11:28.937241+0000 mgr.y (mgr.25390) 110 : cephadm 1 Reconfiguring osd.6 (monmap changed)... 2023-07-10T17:11:30.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:29 smithi116 bash[65980]: audit 2023-07-10T17:11:28.938004+0000 mon.a (mon.0) 207 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-10T17:11:30.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:29 smithi116 bash[65980]: audit 2023-07-10T17:11:28.940266+0000 mon.a (mon.0) 208 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:30.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:29 smithi116 bash[65980]: cephadm 2023-07-10T17:11:28.945890+0000 mgr.y (mgr.25390) 111 : cephadm 1 Reconfiguring daemon osd.6 on smithi192 2023-07-10T17:11:32.381 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:32 smithi192 bash[50362]: cluster 2023-07-10T17:11:30.422256+0000 mgr.y (mgr.25390) 112 : cluster 0 pgmap v61: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:32.381 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:32 smithi192 bash[50362]: audit 2023-07-10T17:11:31.068173+0000 mon.a (mon.0) 209 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:32.381 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:32 smithi192 bash[50362]: audit 2023-07-10T17:11:31.078622+0000 mon.a (mon.0) 210 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:32.382 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:32 smithi192 bash[50362]: cephadm 2023-07-10T17:11:31.115543+0000 mgr.y (mgr.25390) 113 : cephadm 1 Reconfiguring osd.7 (monmap changed)... 2023-07-10T17:11:32.382 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:32 smithi192 bash[50362]: audit 2023-07-10T17:11:31.116210+0000 mon.a (mon.0) 211 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-10T17:11:32.382 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:32 smithi192 bash[50362]: audit 2023-07-10T17:11:31.118134+0000 mon.a (mon.0) 212 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:32.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:32 smithi192 bash[50362]: cephadm 2023-07-10T17:11:31.123677+0000 mgr.y (mgr.25390) 114 : cephadm 1 Reconfiguring daemon osd.7 on smithi192 2023-07-10T17:11:32.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:32 smithi116 bash[63900]: cluster 2023-07-10T17:11:30.422256+0000 mgr.y (mgr.25390) 112 : cluster 0 pgmap v61: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:32.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:32 smithi116 bash[63900]: audit 2023-07-10T17:11:31.068173+0000 mon.a (mon.0) 209 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:32.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:32 smithi116 bash[63900]: audit 2023-07-10T17:11:31.078622+0000 mon.a (mon.0) 210 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:32.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:32 smithi116 bash[63900]: cephadm 2023-07-10T17:11:31.115543+0000 mgr.y (mgr.25390) 113 : cephadm 1 Reconfiguring osd.7 (monmap changed)... 2023-07-10T17:11:32.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:32 smithi116 bash[63900]: audit 2023-07-10T17:11:31.116210+0000 mon.a (mon.0) 211 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-10T17:11:32.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:32 smithi116 bash[63900]: audit 2023-07-10T17:11:31.118134+0000 mon.a (mon.0) 212 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:32.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:32 smithi116 bash[63900]: cephadm 2023-07-10T17:11:31.123677+0000 mgr.y (mgr.25390) 114 : cephadm 1 Reconfiguring daemon osd.7 on smithi192 2023-07-10T17:11:32.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:32 smithi116 bash[65980]: cluster 2023-07-10T17:11:30.422256+0000 mgr.y (mgr.25390) 112 : cluster 0 pgmap v61: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:32.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:32 smithi116 bash[65980]: audit 2023-07-10T17:11:31.068173+0000 mon.a (mon.0) 209 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:32.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:32 smithi116 bash[65980]: audit 2023-07-10T17:11:31.078622+0000 mon.a (mon.0) 210 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:32.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:32 smithi116 bash[65980]: cephadm 2023-07-10T17:11:31.115543+0000 mgr.y (mgr.25390) 113 : cephadm 1 Reconfiguring osd.7 (monmap changed)... 2023-07-10T17:11:32.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:32 smithi116 bash[65980]: audit 2023-07-10T17:11:31.116210+0000 mon.a (mon.0) 211 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-10T17:11:32.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:32 smithi116 bash[65980]: audit 2023-07-10T17:11:31.118134+0000 mon.a (mon.0) 212 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:32.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:32 smithi116 bash[65980]: cephadm 2023-07-10T17:11:31.123677+0000 mgr.y (mgr.25390) 114 : cephadm 1 Reconfiguring daemon osd.7 on smithi192 2023-07-10T17:11:34.201 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:11:34.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:34 smithi192 bash[50362]: cluster 2023-07-10T17:11:32.423922+0000 mgr.y (mgr.25390) 115 : cluster 0 pgmap v62: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:34.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:34 smithi192 bash[50362]: audit 2023-07-10T17:11:33.300664+0000 mon.a (mon.0) 213 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:34.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:34 smithi192 bash[50362]: audit 2023-07-10T17:11:33.309869+0000 mon.a (mon.0) 214 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:34.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:34 smithi192 bash[50362]: audit 2023-07-10T17:11:33.345379+0000 mon.a (mon.0) 215 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.myrhsb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:11:34.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:34 smithi192 bash[50362]: audit 2023-07-10T17:11:33.348354+0000 mon.a (mon.0) 216 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:34.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:34 smithi116 bash[63900]: cluster 2023-07-10T17:11:32.423922+0000 mgr.y (mgr.25390) 115 : cluster 0 pgmap v62: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:34.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:34 smithi116 bash[63900]: audit 2023-07-10T17:11:33.300664+0000 mon.a (mon.0) 213 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:34.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:34 smithi116 bash[63900]: audit 2023-07-10T17:11:33.309869+0000 mon.a (mon.0) 214 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:34.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:34 smithi116 bash[63900]: audit 2023-07-10T17:11:33.345379+0000 mon.a (mon.0) 215 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.myrhsb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:11:34.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:34 smithi116 bash[63900]: audit 2023-07-10T17:11:33.348354+0000 mon.a (mon.0) 216 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:34.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:34 smithi116 bash[65980]: cluster 2023-07-10T17:11:32.423922+0000 mgr.y (mgr.25390) 115 : cluster 0 pgmap v62: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:34.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:34 smithi116 bash[65980]: audit 2023-07-10T17:11:33.300664+0000 mon.a (mon.0) 213 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:34.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:34 smithi116 bash[65980]: audit 2023-07-10T17:11:33.309869+0000 mon.a (mon.0) 214 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:34.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:34 smithi116 bash[65980]: audit 2023-07-10T17:11:33.345379+0000 mon.a (mon.0) 215 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.myrhsb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:11:34.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:34 smithi116 bash[65980]: audit 2023-07-10T17:11:33.348354+0000 mon.a (mon.0) 216 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:34.945 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:11:34.945 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (4m) 80s ago 11m 10.8M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:11:34.945 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (3m) 49s ago 10m 25.6M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:11:34.946 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 error 80s ago 8m - - 2023-07-10T17:11:34.946 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (2m) 49s ago 15m 428M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:11:34.946 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (3m) 80s ago 17m 464M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:11:34.946 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (2m) 80s ago 17m 45.2M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:11:34.946 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (63s) 49s ago 15m 40.2M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 74a082a2e795 2023-07-10T17:11:34.946 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (89s) 80s ago 15m 38.1M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:11:34.946 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (4m) 80s ago 11m 9287k - 0.18.1 e5a616e4b9cf 3bd058d91353 2023-07-10T17:11:34.947 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (4m) 49s ago 11m 3607k - 0.18.1 e5a616e4b9cf 63551615c679 2023-07-10T17:11:34.947 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (14m) 80s ago 14m 55.4M 4096M 16.2.0 24ecd6d5f14c 4fa26367c830 2023-07-10T17:11:34.947 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (14m) 80s ago 14m 53.6M 4096M 16.2.0 24ecd6d5f14c c157509b3fd4 2023-07-10T17:11:34.947 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (14m) 80s ago 14m 49.4M 4096M 16.2.0 24ecd6d5f14c f1fd77b3c454 2023-07-10T17:11:34.947 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (13m) 80s ago 13m 54.9M 4096M 16.2.0 24ecd6d5f14c 60949d479c11 2023-07-10T17:11:34.947 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (13m) 49s ago 13m 55.1M 4096M 16.2.0 24ecd6d5f14c 75fc0b36aab8 2023-07-10T17:11:34.947 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (12m) 49s ago 12m 52.8M 4096M 16.2.0 24ecd6d5f14c d673065333dd 2023-07-10T17:11:34.948 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (12m) 49s ago 12m 53.7M 4096M 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:11:34.948 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (12m) 49s ago 12m 56.7M 4096M 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:11:34.948 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 error 49s ago 11m - - 2023-07-10T17:11:34.948 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (8m) 80s ago 8m 54.7M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:11:34.948 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (9m) 49s ago 9m 54.8M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:11:34.948 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (8m) 80s ago 8m 54.6M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:11:34.948 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (8m) 49s ago 8m 56.6M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:11:35.380 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:11:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:11:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:11:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:11:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:11:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:11:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:11:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:11:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2023-07-10T17:11:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:11:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:11:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:11:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:11:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:11:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 12, 2023-07-10T17:11:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 5 2023-07-10T17:11:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:11:35.384 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:11:35.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:35 smithi192 bash[50362]: cephadm 2023-07-10T17:11:33.344748+0000 mgr.y (mgr.25390) 116 : cephadm 1 Reconfiguring rgw.foo.smithi192.myrhsb (monmap changed)... 2023-07-10T17:11:35.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:35 smithi192 bash[50362]: cephadm 2023-07-10T17:11:33.349908+0000 mgr.y (mgr.25390) 117 : cephadm 1 Reconfiguring daemon rgw.foo.smithi192.myrhsb on smithi192 2023-07-10T17:11:35.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:35 smithi192 bash[50362]: audit 2023-07-10T17:11:34.176433+0000 mgr.y (mgr.25390) 118 : audit 0 from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:35.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:35 smithi116 bash[65980]: cephadm 2023-07-10T17:11:33.344748+0000 mgr.y (mgr.25390) 116 : cephadm 1 Reconfiguring rgw.foo.smithi192.myrhsb (monmap changed)... 2023-07-10T17:11:35.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:35 smithi116 bash[65980]: cephadm 2023-07-10T17:11:33.349908+0000 mgr.y (mgr.25390) 117 : cephadm 1 Reconfiguring daemon rgw.foo.smithi192.myrhsb on smithi192 2023-07-10T17:11:35.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:35 smithi116 bash[65980]: audit 2023-07-10T17:11:34.176433+0000 mgr.y (mgr.25390) 118 : audit 0 from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:35.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:35 smithi116 bash[63900]: cephadm 2023-07-10T17:11:33.344748+0000 mgr.y (mgr.25390) 116 : cephadm 1 Reconfiguring rgw.foo.smithi192.myrhsb (monmap changed)... 2023-07-10T17:11:35.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:35 smithi116 bash[63900]: cephadm 2023-07-10T17:11:33.349908+0000 mgr.y (mgr.25390) 117 : cephadm 1 Reconfiguring daemon rgw.foo.smithi192.myrhsb on smithi192 2023-07-10T17:11:35.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:35 smithi116 bash[63900]: audit 2023-07-10T17:11:34.176433+0000 mgr.y (mgr.25390) 118 : audit 0 from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:35.747 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:11:35.748 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:11:35.748 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:11:35.748 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:11:35.748 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:11:35.748 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2023-07-10T17:11:35.748 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2023-07-10T17:11:35.748 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:11:35.749 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "5/23 daemons upgraded", 2023-07-10T17:11:35.749 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading mon daemons", 2023-07-10T17:11:35.749 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:11:35.749 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:11:36.176 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:11:36.176 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:11:36.177 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:11:36.177 INFO:teuthology.orchestra.run.smithi116.stdout: daemon prometheus.a on smithi192 is in error state 2023-07-10T17:11:36.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:36 smithi192 bash[50362]: cluster 2023-07-10T17:11:34.424683+0000 mgr.y (mgr.25390) 119 : cluster 0 pgmap v63: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:36.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:36 smithi192 bash[50362]: audit 2023-07-10T17:11:34.554360+0000 mgr.y (mgr.25390) 120 : audit 0 from='client.34419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:36.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:36 smithi192 bash[50362]: audit 2023-07-10T17:11:34.918773+0000 mgr.y (mgr.25390) 121 : audit 0 from='client.44281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:36.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:36 smithi192 bash[50362]: audit 2023-07-10T17:11:35.375921+0000 mon.c (mon.1) 12 : audit 0 from='client.44287 172.21.15.116:0/1130033188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:11:36.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:36 smithi192 bash[50362]: audit 2023-07-10T17:11:35.457255+0000 mon.a (mon.0) 217 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:36.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:36 smithi192 bash[50362]: audit 2023-07-10T17:11:35.467585+0000 mon.a (mon.0) 218 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:36.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:36 smithi192 bash[50362]: audit 2023-07-10T17:11:35.469695+0000 mon.a (mon.0) 219 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.yqkmva", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:11:36.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:36 smithi192 bash[50362]: audit 2023-07-10T17:11:35.472307+0000 mon.a (mon.0) 220 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:36.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:36 smithi116 bash[63900]: cluster 2023-07-10T17:11:34.424683+0000 mgr.y (mgr.25390) 119 : cluster 0 pgmap v63: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:36.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:36 smithi116 bash[63900]: audit 2023-07-10T17:11:34.554360+0000 mgr.y (mgr.25390) 120 : audit 0 from='client.34419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:36.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:36 smithi116 bash[63900]: audit 2023-07-10T17:11:34.918773+0000 mgr.y (mgr.25390) 121 : audit 0 from='client.44281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:36.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:36 smithi116 bash[63900]: audit 2023-07-10T17:11:35.375921+0000 mon.c (mon.1) 12 : audit 0 from='client.44287 172.21.15.116:0/1130033188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:11:36.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:36 smithi116 bash[63900]: audit 2023-07-10T17:11:35.457255+0000 mon.a (mon.0) 217 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:36.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:36 smithi116 bash[63900]: audit 2023-07-10T17:11:35.467585+0000 mon.a (mon.0) 218 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:36.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:36 smithi116 bash[63900]: audit 2023-07-10T17:11:35.469695+0000 mon.a (mon.0) 219 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.yqkmva", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:11:36.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:36 smithi116 bash[63900]: audit 2023-07-10T17:11:35.472307+0000 mon.a (mon.0) 220 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:36.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:36 smithi116 bash[65980]: cluster 2023-07-10T17:11:34.424683+0000 mgr.y (mgr.25390) 119 : cluster 0 pgmap v63: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:36.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:36 smithi116 bash[65980]: audit 2023-07-10T17:11:34.554360+0000 mgr.y (mgr.25390) 120 : audit 0 from='client.34419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:36.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:36 smithi116 bash[65980]: audit 2023-07-10T17:11:34.918773+0000 mgr.y (mgr.25390) 121 : audit 0 from='client.44281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:36.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:36 smithi116 bash[65980]: audit 2023-07-10T17:11:35.375921+0000 mon.c (mon.1) 12 : audit 0 from='client.44287 172.21.15.116:0/1130033188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:11:36.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:36 smithi116 bash[65980]: audit 2023-07-10T17:11:35.457255+0000 mon.a (mon.0) 217 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:36.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:36 smithi116 bash[65980]: audit 2023-07-10T17:11:35.467585+0000 mon.a (mon.0) 218 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:36.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:36 smithi116 bash[65980]: audit 2023-07-10T17:11:35.469695+0000 mon.a (mon.0) 219 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.yqkmva", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:11:36.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:36 smithi116 bash[65980]: audit 2023-07-10T17:11:35.472307+0000 mon.a (mon.0) 220 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:37.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:37 smithi192 bash[50362]: cephadm 2023-07-10T17:11:35.469158+0000 mgr.y (mgr.25390) 122 : cephadm 1 Reconfiguring rgw.smpl.smithi192.yqkmva (monmap changed)... 2023-07-10T17:11:37.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:37 smithi192 bash[50362]: cephadm 2023-07-10T17:11:35.473761+0000 mgr.y (mgr.25390) 123 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi192.yqkmva on smithi192 2023-07-10T17:11:37.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:37 smithi192 bash[50362]: audit 2023-07-10T17:11:35.742911+0000 mgr.y (mgr.25390) 124 : audit 0 from='client.34437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:37.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:37 smithi192 bash[50362]: audit 2023-07-10T17:11:36.172227+0000 mon.c (mon.1) 13 : audit 0 from='client.34443 172.21.15.116:0/1478590593' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:11:37.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:37 smithi116 bash[63900]: cephadm 2023-07-10T17:11:35.469158+0000 mgr.y (mgr.25390) 122 : cephadm 1 Reconfiguring rgw.smpl.smithi192.yqkmva (monmap changed)... 2023-07-10T17:11:37.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:37 smithi116 bash[63900]: cephadm 2023-07-10T17:11:35.473761+0000 mgr.y (mgr.25390) 123 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi192.yqkmva on smithi192 2023-07-10T17:11:37.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:37 smithi116 bash[63900]: audit 2023-07-10T17:11:35.742911+0000 mgr.y (mgr.25390) 124 : audit 0 from='client.34437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:37.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:37 smithi116 bash[63900]: audit 2023-07-10T17:11:36.172227+0000 mon.c (mon.1) 13 : audit 0 from='client.34443 172.21.15.116:0/1478590593' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:11:37.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:37 smithi116 bash[65980]: cephadm 2023-07-10T17:11:35.469158+0000 mgr.y (mgr.25390) 122 : cephadm 1 Reconfiguring rgw.smpl.smithi192.yqkmva (monmap changed)... 2023-07-10T17:11:37.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:37 smithi116 bash[65980]: cephadm 2023-07-10T17:11:35.473761+0000 mgr.y (mgr.25390) 123 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi192.yqkmva on smithi192 2023-07-10T17:11:37.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:37 smithi116 bash[65980]: audit 2023-07-10T17:11:35.742911+0000 mgr.y (mgr.25390) 124 : audit 0 from='client.34437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:11:37.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:37 smithi116 bash[65980]: audit 2023-07-10T17:11:36.172227+0000 mon.c (mon.1) 13 : audit 0 from='client.34443 172.21.15.116:0/1478590593' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:11:38.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:38 smithi192 bash[50362]: cluster 2023-07-10T17:11:36.425408+0000 mgr.y (mgr.25390) 125 : cluster 0 pgmap v64: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:38.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:38 smithi192 bash[50362]: audit 2023-07-10T17:11:37.717289+0000 mon.a (mon.0) 221 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:38.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:38 smithi192 bash[50362]: audit 2023-07-10T17:11:37.728167+0000 mon.a (mon.0) 222 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:38.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:38 smithi192 bash[50362]: audit 2023-07-10T17:11:37.752845+0000 mon.a (mon.0) 223 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:11:38.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:38 smithi192 bash[50362]: audit 2023-07-10T17:11:37.757181+0000 mon.a (mon.0) 224 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:11:38.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:38 smithi192 bash[50362]: audit 2023-07-10T17:11:37.759871+0000 mon.a (mon.0) 225 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:11:38.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:38 smithi192 bash[50362]: audit 2023-07-10T17:11:37.769098+0000 mon.a (mon.0) 226 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:38.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:38 smithi192 bash[50362]: audit 2023-07-10T17:11:37.770821+0000 mon.a (mon.0) 227 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2023-07-10T17:11:38.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:38 smithi192 bash[50362]: audit 2023-07-10T17:11:37.775587+0000 mon.a (mon.0) 228 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2023-07-10T17:11:38.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:38 smithi192 bash[50362]: audit 2023-07-10T17:11:37.776888+0000 mon.a (mon.0) 229 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2023-07-10T17:11:38.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:38 smithi192 bash[50362]: audit 2023-07-10T17:11:37.782949+0000 mon.a (mon.0) 230 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2023-07-10T17:11:38.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:38 smithi192 bash[50362]: audit 2023-07-10T17:11:37.784543+0000 mon.a (mon.0) 231 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2023-07-10T17:11:38.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:38 smithi192 bash[50362]: audit 2023-07-10T17:11:37.792124+0000 mon.a (mon.0) 232 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2023-07-10T17:11:38.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:38 smithi192 bash[50362]: audit 2023-07-10T17:11:37.794037+0000 mon.a (mon.0) 233 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:11:38.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:38 smithi192 bash[50362]: audit 2023-07-10T17:11:37.801421+0000 mon.a (mon.0) 234 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:38.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:38 smithi192 bash[50362]: audit 2023-07-10T17:11:37.803939+0000 mon.a (mon.0) 235 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-07-10T17:11:38.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[63900]: cluster 2023-07-10T17:11:36.425408+0000 mgr.y (mgr.25390) 125 : cluster 0 pgmap v64: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:38.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[63900]: audit 2023-07-10T17:11:37.717289+0000 mon.a (mon.0) 221 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:38.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[63900]: audit 2023-07-10T17:11:37.728167+0000 mon.a (mon.0) 222 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:38.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[63900]: audit 2023-07-10T17:11:37.752845+0000 mon.a (mon.0) 223 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:11:38.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[63900]: audit 2023-07-10T17:11:37.757181+0000 mon.a (mon.0) 224 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:11:38.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[63900]: audit 2023-07-10T17:11:37.759871+0000 mon.a (mon.0) 225 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:11:38.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[63900]: audit 2023-07-10T17:11:37.769098+0000 mon.a (mon.0) 226 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:38.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[63900]: audit 2023-07-10T17:11:37.770821+0000 mon.a (mon.0) 227 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2023-07-10T17:11:38.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[63900]: audit 2023-07-10T17:11:37.775587+0000 mon.a (mon.0) 228 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2023-07-10T17:11:38.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[63900]: audit 2023-07-10T17:11:37.776888+0000 mon.a (mon.0) 229 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2023-07-10T17:11:38.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[63900]: audit 2023-07-10T17:11:37.782949+0000 mon.a (mon.0) 230 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2023-07-10T17:11:38.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[63900]: audit 2023-07-10T17:11:37.784543+0000 mon.a (mon.0) 231 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2023-07-10T17:11:38.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[63900]: audit 2023-07-10T17:11:37.792124+0000 mon.a (mon.0) 232 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2023-07-10T17:11:38.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[63900]: audit 2023-07-10T17:11:37.794037+0000 mon.a (mon.0) 233 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:11:38.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[63900]: audit 2023-07-10T17:11:37.801421+0000 mon.a (mon.0) 234 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:38.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[63900]: audit 2023-07-10T17:11:37.803939+0000 mon.a (mon.0) 235 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-07-10T17:11:38.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[65980]: cluster 2023-07-10T17:11:36.425408+0000 mgr.y (mgr.25390) 125 : cluster 0 pgmap v64: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:38.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[65980]: audit 2023-07-10T17:11:37.717289+0000 mon.a (mon.0) 221 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:38.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[65980]: audit 2023-07-10T17:11:37.728167+0000 mon.a (mon.0) 222 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:38.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[65980]: audit 2023-07-10T17:11:37.752845+0000 mon.a (mon.0) 223 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:11:38.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[65980]: audit 2023-07-10T17:11:37.757181+0000 mon.a (mon.0) 224 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:11:38.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[65980]: audit 2023-07-10T17:11:37.759871+0000 mon.a (mon.0) 225 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:11:38.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[65980]: audit 2023-07-10T17:11:37.769098+0000 mon.a (mon.0) 226 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:38.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[65980]: audit 2023-07-10T17:11:37.770821+0000 mon.a (mon.0) 227 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2023-07-10T17:11:38.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[65980]: audit 2023-07-10T17:11:37.775587+0000 mon.a (mon.0) 228 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2023-07-10T17:11:38.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[65980]: audit 2023-07-10T17:11:37.776888+0000 mon.a (mon.0) 229 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2023-07-10T17:11:38.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[65980]: audit 2023-07-10T17:11:37.782949+0000 mon.a (mon.0) 230 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2023-07-10T17:11:38.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[65980]: audit 2023-07-10T17:11:37.784543+0000 mon.a (mon.0) 231 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2023-07-10T17:11:38.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[65980]: audit 2023-07-10T17:11:37.792124+0000 mon.a (mon.0) 232 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2023-07-10T17:11:38.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[65980]: audit 2023-07-10T17:11:37.794037+0000 mon.a (mon.0) 233 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:11:38.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[65980]: audit 2023-07-10T17:11:37.801421+0000 mon.a (mon.0) 234 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:38.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:38 smithi116 bash[65980]: audit 2023-07-10T17:11:37.803939+0000 mon.a (mon.0) 235 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-07-10T17:11:39.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:39 smithi192 bash[50362]: cephadm 2023-07-10T17:11:37.761031+0000 mgr.y (mgr.25390) 126 : cephadm 1 Upgrade: Setting container_image for all mon 2023-07-10T17:11:39.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:39 smithi192 bash[50362]: cephadm 2023-07-10T17:11:37.795148+0000 mgr.y (mgr.25390) 127 : cephadm 1 Upgrade: Setting container_image for all crash 2023-07-10T17:11:39.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:39 smithi192 bash[50362]: audit 2023-07-10T17:11:37.804237+0000 mgr.y (mgr.25390) 128 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-07-10T17:11:39.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:39 smithi192 bash[50362]: cephadm 2023-07-10T17:11:37.806058+0000 mgr.y (mgr.25390) 129 : cephadm 1 Upgrade: osd.0 is safe to restart 2023-07-10T17:11:39.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:39 smithi116 bash[63900]: cephadm 2023-07-10T17:11:37.761031+0000 mgr.y (mgr.25390) 126 : cephadm 1 Upgrade: Setting container_image for all mon 2023-07-10T17:11:39.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:39 smithi116 bash[63900]: cephadm 2023-07-10T17:11:37.795148+0000 mgr.y (mgr.25390) 127 : cephadm 1 Upgrade: Setting container_image for all crash 2023-07-10T17:11:39.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:39 smithi116 bash[63900]: audit 2023-07-10T17:11:37.804237+0000 mgr.y (mgr.25390) 128 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-07-10T17:11:39.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:39 smithi116 bash[63900]: cephadm 2023-07-10T17:11:37.806058+0000 mgr.y (mgr.25390) 129 : cephadm 1 Upgrade: osd.0 is safe to restart 2023-07-10T17:11:39.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:39 smithi116 bash[65980]: cephadm 2023-07-10T17:11:37.761031+0000 mgr.y (mgr.25390) 126 : cephadm 1 Upgrade: Setting container_image for all mon 2023-07-10T17:11:39.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:39 smithi116 bash[65980]: cephadm 2023-07-10T17:11:37.795148+0000 mgr.y (mgr.25390) 127 : cephadm 1 Upgrade: Setting container_image for all crash 2023-07-10T17:11:39.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:39 smithi116 bash[65980]: audit 2023-07-10T17:11:37.804237+0000 mgr.y (mgr.25390) 128 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-07-10T17:11:39.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:39 smithi116 bash[65980]: cephadm 2023-07-10T17:11:37.806058+0000 mgr.y (mgr.25390) 129 : cephadm 1 Upgrade: osd.0 is safe to restart 2023-07-10T17:11:40.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:40 smithi192 bash[50362]: cluster 2023-07-10T17:11:38.427079+0000 mgr.y (mgr.25390) 130 : cluster 0 pgmap v65: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:40.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:40 smithi116 bash[63900]: cluster 2023-07-10T17:11:38.427079+0000 mgr.y (mgr.25390) 130 : cluster 0 pgmap v65: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:40.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:40 smithi116 bash[65980]: cluster 2023-07-10T17:11:38.427079+0000 mgr.y (mgr.25390) 130 : cluster 0 pgmap v65: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:41.549 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:41 smithi116 bash[63900]: cephadm 2023-07-10T17:11:40.176420+0000 mgr.y (mgr.25390) 131 : cephadm 1 Upgrade: Updating osd.0 2023-07-10T17:11:41.549 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:41 smithi116 bash[63900]: audit 2023-07-10T17:11:40.187335+0000 mon.a (mon.0) 236 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:41.549 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:41 smithi116 bash[63900]: audit 2023-07-10T17:11:40.189101+0000 mon.a (mon.0) 237 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-10T17:11:41.549 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:41 smithi116 bash[63900]: audit 2023-07-10T17:11:40.190563+0000 mon.a (mon.0) 238 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:41.549 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:41 smithi116 bash[63900]: cephadm 2023-07-10T17:11:40.193762+0000 mgr.y (mgr.25390) 132 : cephadm 1 Deploying daemon osd.0 on smithi116 2023-07-10T17:11:41.550 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:41 smithi116 bash[65980]: cephadm 2023-07-10T17:11:40.176420+0000 mgr.y (mgr.25390) 131 : cephadm 1 Upgrade: Updating osd.0 2023-07-10T17:11:41.550 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:41 smithi116 bash[65980]: audit 2023-07-10T17:11:40.187335+0000 mon.a (mon.0) 236 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:41.550 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:41 smithi116 bash[65980]: audit 2023-07-10T17:11:40.189101+0000 mon.a (mon.0) 237 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-10T17:11:41.550 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:41 smithi116 bash[65980]: audit 2023-07-10T17:11:40.190563+0000 mon.a (mon.0) 238 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:41.550 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:41 smithi116 bash[65980]: cephadm 2023-07-10T17:11:40.193762+0000 mgr.y (mgr.25390) 132 : cephadm 1 Deploying daemon osd.0 on smithi116 2023-07-10T17:11:41.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:41 smithi192 bash[50362]: cephadm 2023-07-10T17:11:40.176420+0000 mgr.y (mgr.25390) 131 : cephadm 1 Upgrade: Updating osd.0 2023-07-10T17:11:41.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:41 smithi192 bash[50362]: audit 2023-07-10T17:11:40.187335+0000 mon.a (mon.0) 236 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:41.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:41 smithi192 bash[50362]: audit 2023-07-10T17:11:40.189101+0000 mon.a (mon.0) 237 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-10T17:11:41.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:41 smithi192 bash[50362]: audit 2023-07-10T17:11:40.190563+0000 mon.a (mon.0) 238 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:11:41.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:41 smithi192 bash[50362]: cephadm 2023-07-10T17:11:40.193762+0000 mgr.y (mgr.25390) 132 : cephadm 1 Deploying daemon osd.0 on smithi116 2023-07-10T17:11:42.589 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:42 smithi116 bash[63900]: cluster 2023-07-10T17:11:40.427777+0000 mgr.y (mgr.25390) 133 : cluster 0 pgmap v66: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:42.590 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:42 smithi116 bash[65980]: cluster 2023-07-10T17:11:40.427777+0000 mgr.y (mgr.25390) 133 : cluster 0 pgmap v66: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:42.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:42 smithi192 bash[50362]: cluster 2023-07-10T17:11:40.427777+0000 mgr.y (mgr.25390) 133 : cluster 0 pgmap v66: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:44.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:44 smithi116 bash[63900]: cluster 2023-07-10T17:11:42.429356+0000 mgr.y (mgr.25390) 134 : cluster 0 pgmap v67: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:44.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:44 smithi116 bash[65980]: cluster 2023-07-10T17:11:42.429356+0000 mgr.y (mgr.25390) 134 : cluster 0 pgmap v67: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:44.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:44 smithi192 bash[50362]: cluster 2023-07-10T17:11:42.429356+0000 mgr.y (mgr.25390) 134 : cluster 0 pgmap v67: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:45.143 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:45 smithi116 systemd[1]: Stopping Ceph osd.0 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:11:45.643 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:45 smithi116 bash[30981]: debug 2023-07-10T17:11:45.151+0000 7f8b9208c700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:11:45.644 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:45 smithi116 bash[30981]: debug 2023-07-10T17:11:45.151+0000 7f8b9208c700 -1 osd.0 77 *** Got signal Terminated *** 2023-07-10T17:11:45.644 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:45 smithi116 bash[30981]: debug 2023-07-10T17:11:45.151+0000 7f8b9208c700 -1 osd.0 77 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-10T17:11:46.393 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[69925]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-osd.0 2023-07-10T17:11:46.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:44.430095+0000 mgr.y (mgr.25390) 135 : cluster 0 pgmap v68: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:46.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.435989+0000 mon.a (mon.0) 239 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-10T17:11:46.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.436087+0000 mon.a (mon.0) 240 : cluster 1 osd.0 failed (root=default,host=smithi116) (connection refused reported by osd.5) 2023-07-10T17:11:46.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.436602+0000 mon.a (mon.0) 241 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-10T17:11:46.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.436995+0000 mon.a (mon.0) 242 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-10T17:11:46.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.437278+0000 mon.a (mon.0) 243 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-10T17:11:46.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.437549+0000 mon.a (mon.0) 244 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-10T17:11:46.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.437636+0000 mon.a (mon.0) 245 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-10T17:11:46.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.437806+0000 mon.a (mon.0) 246 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-10T17:11:46.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.437881+0000 mon.a (mon.0) 247 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-10T17:11:46.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.438148+0000 mon.a (mon.0) 248 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-10T17:11:46.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.438230+0000 mon.a (mon.0) 249 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-10T17:11:46.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.438475+0000 mon.a (mon.0) 250 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-10T17:11:46.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.438561+0000 mon.a (mon.0) 251 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-10T17:11:46.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.438807+0000 mon.a (mon.0) 252 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-10T17:11:46.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.438977+0000 mon.a (mon.0) 253 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-10T17:11:46.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.839213+0000 mon.a (mon.0) 254 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-10T17:11:46.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.839711+0000 mon.a (mon.0) 255 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-10T17:11:46.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.840485+0000 mon.a (mon.0) 256 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-10T17:11:46.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.840976+0000 mon.a (mon.0) 257 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-10T17:11:46.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.841367+0000 mon.a (mon.0) 258 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-10T17:11:46.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.841894+0000 mon.a (mon.0) 259 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-10T17:11:46.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.842248+0000 mon.a (mon.0) 260 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-10T17:11:46.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.842626+0000 mon.a (mon.0) 261 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-10T17:11:46.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.843000+0000 mon.a (mon.0) 262 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-10T17:11:46.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.843558+0000 mon.a (mon.0) 263 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-10T17:11:46.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:44.430095+0000 mgr.y (mgr.25390) 135 : cluster 0 pgmap v68: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:46.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.435989+0000 mon.a (mon.0) 239 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-10T17:11:46.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.436087+0000 mon.a (mon.0) 240 : cluster 1 osd.0 failed (root=default,host=smithi116) (connection refused reported by osd.5) 2023-07-10T17:11:46.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.436602+0000 mon.a (mon.0) 241 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-10T17:11:46.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.436995+0000 mon.a (mon.0) 242 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-10T17:11:46.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.437278+0000 mon.a (mon.0) 243 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-10T17:11:46.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.437549+0000 mon.a (mon.0) 244 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-10T17:11:46.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.437636+0000 mon.a (mon.0) 245 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-10T17:11:46.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.437806+0000 mon.a (mon.0) 246 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-10T17:11:46.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.437881+0000 mon.a (mon.0) 247 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-10T17:11:46.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.438148+0000 mon.a (mon.0) 248 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-10T17:11:46.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.438230+0000 mon.a (mon.0) 249 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-10T17:11:46.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.438475+0000 mon.a (mon.0) 250 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-10T17:11:46.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.438561+0000 mon.a (mon.0) 251 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-10T17:11:46.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.438807+0000 mon.a (mon.0) 252 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-10T17:11:46.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.438977+0000 mon.a (mon.0) 253 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-10T17:11:46.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.839213+0000 mon.a (mon.0) 254 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-10T17:11:46.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.839711+0000 mon.a (mon.0) 255 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-10T17:11:46.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.840485+0000 mon.a (mon.0) 256 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-10T17:11:46.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.840976+0000 mon.a (mon.0) 257 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-10T17:11:46.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.841367+0000 mon.a (mon.0) 258 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-10T17:11:46.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.841894+0000 mon.a (mon.0) 259 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-10T17:11:46.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.842248+0000 mon.a (mon.0) 260 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-10T17:11:46.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.842626+0000 mon.a (mon.0) 261 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-10T17:11:46.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.843000+0000 mon.a (mon.0) 262 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-10T17:11:46.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.843558+0000 mon.a (mon.0) 263 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-10T17:11:46.401 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.843786+0000 mon.a (mon.0) 264 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-10T17:11:46.402 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.844258+0000 mon.a (mon.0) 265 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-10T17:11:46.402 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.844609+0000 mon.a (mon.0) 266 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-10T17:11:46.402 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[63900]: cluster 2023-07-10T17:11:45.844844+0000 mon.a (mon.0) 267 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-10T17:11:46.402 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.843786+0000 mon.a (mon.0) 264 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-10T17:11:46.403 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.844258+0000 mon.a (mon.0) 265 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-10T17:11:46.403 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.844609+0000 mon.a (mon.0) 266 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-10T17:11:46.403 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:46 smithi116 bash[65980]: cluster 2023-07-10T17:11:45.844844+0000 mon.a (mon.0) 267 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-10T17:11:46.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:44.430095+0000 mgr.y (mgr.25390) 135 : cluster 0 pgmap v68: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:46.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.435989+0000 mon.a (mon.0) 239 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-10T17:11:46.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.436087+0000 mon.a (mon.0) 240 : cluster 1 osd.0 failed (root=default,host=smithi116) (connection refused reported by osd.5) 2023-07-10T17:11:46.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.436602+0000 mon.a (mon.0) 241 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-10T17:11:46.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.436995+0000 mon.a (mon.0) 242 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-10T17:11:46.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.437278+0000 mon.a (mon.0) 243 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-10T17:11:46.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.437549+0000 mon.a (mon.0) 244 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-10T17:11:46.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.437636+0000 mon.a (mon.0) 245 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-10T17:11:46.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.437806+0000 mon.a (mon.0) 246 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-10T17:11:46.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.437881+0000 mon.a (mon.0) 247 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-10T17:11:46.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.438148+0000 mon.a (mon.0) 248 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-10T17:11:46.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.438230+0000 mon.a (mon.0) 249 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-10T17:11:46.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.438475+0000 mon.a (mon.0) 250 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-10T17:11:46.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.438561+0000 mon.a (mon.0) 251 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-10T17:11:46.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.438807+0000 mon.a (mon.0) 252 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-10T17:11:46.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.438977+0000 mon.a (mon.0) 253 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-10T17:11:46.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.839213+0000 mon.a (mon.0) 254 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-10T17:11:46.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.839711+0000 mon.a (mon.0) 255 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-10T17:11:46.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.840485+0000 mon.a (mon.0) 256 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-10T17:11:46.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.840976+0000 mon.a (mon.0) 257 : cluster 0 osd.0 reported immediately failed by osd.5 2023-07-10T17:11:46.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.841367+0000 mon.a (mon.0) 258 : cluster 0 osd.0 reported immediately failed by osd.4 2023-07-10T17:11:46.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.841894+0000 mon.a (mon.0) 259 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-10T17:11:46.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.842248+0000 mon.a (mon.0) 260 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-10T17:11:46.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.842626+0000 mon.a (mon.0) 261 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-10T17:11:46.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.843000+0000 mon.a (mon.0) 262 : cluster 0 osd.0 reported immediately failed by osd.7 2023-07-10T17:11:46.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.843558+0000 mon.a (mon.0) 263 : cluster 0 osd.0 reported immediately failed by osd.6 2023-07-10T17:11:46.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.843786+0000 mon.a (mon.0) 264 : cluster 0 osd.0 reported immediately failed by osd.2 2023-07-10T17:11:46.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.844258+0000 mon.a (mon.0) 265 : cluster 0 osd.0 reported immediately failed by osd.3 2023-07-10T17:11:46.642 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.844609+0000 mon.a (mon.0) 266 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-10T17:11:46.642 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:46 smithi192 bash[50362]: cluster 2023-07-10T17:11:45.844844+0000 mon.a (mon.0) 267 : cluster 0 osd.0 reported immediately failed by osd.1 2023-07-10T17:11:47.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:47 smithi192 bash[50362]: cluster 2023-07-10T17:11:46.199597+0000 mon.a (mon.0) 268 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:11:47.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:47 smithi192 bash[50362]: cluster 2023-07-10T17:11:46.212675+0000 osd.1 (osd.1) 3 : cluster 3 failed to encode map e78 with expected crc 2023-07-10T17:11:47.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:47 smithi192 bash[50362]: cluster 2023-07-10T17:11:46.212820+0000 osd.7 (osd.7) 4 : cluster 3 failed to encode map e78 with expected crc 2023-07-10T17:11:47.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:47 smithi192 bash[50362]: cluster 2023-07-10T17:11:46.214407+0000 mon.a (mon.0) 269 : cluster 0 osdmap e78: 8 total, 7 up, 8 in 2023-07-10T17:11:47.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:47 smithi192 bash[50362]: cluster 2023-07-10T17:11:46.215362+0000 osd.3 (osd.3) 4 : cluster 3 failed to encode map e78 with expected crc 2023-07-10T17:11:47.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:47 smithi192 bash[50362]: cluster 2023-07-10T17:11:46.216427+0000 osd.7 (osd.7) 5 : cluster 3 failed to encode map e78 with expected crc 2023-07-10T17:11:47.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:47 smithi192 bash[50362]: cluster 2023-07-10T17:11:46.216459+0000 osd.5 (osd.5) 4 : cluster 3 failed to encode map e78 with expected crc 2023-07-10T17:11:47.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:47 smithi116 bash[63900]: cluster 2023-07-10T17:11:46.199597+0000 mon.a (mon.0) 268 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:11:47.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:47 smithi116 bash[63900]: cluster 2023-07-10T17:11:46.212675+0000 osd.1 (osd.1) 3 : cluster 3 failed to encode map e78 with expected crc 2023-07-10T17:11:47.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:47 smithi116 bash[63900]: cluster 2023-07-10T17:11:46.212820+0000 osd.7 (osd.7) 4 : cluster 3 failed to encode map e78 with expected crc 2023-07-10T17:11:47.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:47 smithi116 bash[63900]: cluster 2023-07-10T17:11:46.214407+0000 mon.a (mon.0) 269 : cluster 0 osdmap e78: 8 total, 7 up, 8 in 2023-07-10T17:11:47.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:47 smithi116 bash[63900]: cluster 2023-07-10T17:11:46.215362+0000 osd.3 (osd.3) 4 : cluster 3 failed to encode map e78 with expected crc 2023-07-10T17:11:47.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:47 smithi116 bash[63900]: cluster 2023-07-10T17:11:46.216427+0000 osd.7 (osd.7) 5 : cluster 3 failed to encode map e78 with expected crc 2023-07-10T17:11:47.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:47 smithi116 bash[63900]: cluster 2023-07-10T17:11:46.216459+0000 osd.5 (osd.5) 4 : cluster 3 failed to encode map e78 with expected crc 2023-07-10T17:11:47.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:47 smithi116 bash[65980]: cluster 2023-07-10T17:11:46.199597+0000 mon.a (mon.0) 268 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:11:47.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:47 smithi116 bash[65980]: cluster 2023-07-10T17:11:46.212675+0000 osd.1 (osd.1) 3 : cluster 3 failed to encode map e78 with expected crc 2023-07-10T17:11:47.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:47 smithi116 bash[65980]: cluster 2023-07-10T17:11:46.212820+0000 osd.7 (osd.7) 4 : cluster 3 failed to encode map e78 with expected crc 2023-07-10T17:11:47.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:47 smithi116 bash[65980]: cluster 2023-07-10T17:11:46.214407+0000 mon.a (mon.0) 269 : cluster 0 osdmap e78: 8 total, 7 up, 8 in 2023-07-10T17:11:47.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:47 smithi116 bash[65980]: cluster 2023-07-10T17:11:46.215362+0000 osd.3 (osd.3) 4 : cluster 3 failed to encode map e78 with expected crc 2023-07-10T17:11:47.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:47 smithi116 bash[65980]: cluster 2023-07-10T17:11:46.216427+0000 osd.7 (osd.7) 5 : cluster 3 failed to encode map e78 with expected crc 2023-07-10T17:11:47.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:47 smithi116 bash[65980]: cluster 2023-07-10T17:11:46.216459+0000 osd.5 (osd.5) 4 : cluster 3 failed to encode map e78 with expected crc 2023-07-10T17:11:48.471 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:48 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.0.service: Succeeded. 2023-07-10T17:11:48.472 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:48 smithi116 systemd[1]: Stopped Ceph osd.0 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:11:48.472 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[63900]: cluster 2023-07-10T17:11:46.430810+0000 mgr.y (mgr.25390) 136 : cluster 0 pgmap v70: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:48.472 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[63900]: cluster 2023-07-10T17:11:47.219428+0000 osd.7 (osd.7) 6 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.472 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[63900]: cluster 2023-07-10T17:11:47.220420+0000 mon.a (mon.0) 270 : cluster 0 osdmap e79: 8 total, 7 up, 8 in 2023-07-10T17:11:48.473 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[63900]: cluster 2023-07-10T17:11:47.220971+0000 osd.3 (osd.3) 5 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.473 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[63900]: cluster 2023-07-10T17:11:47.221704+0000 osd.4 (osd.4) 4 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.473 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[63900]: cluster 2023-07-10T17:11:47.221935+0000 osd.5 (osd.5) 5 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.473 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[63900]: cluster 2023-07-10T17:11:47.222614+0000 osd.6 (osd.6) 3 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.473 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[63900]: cluster 2023-07-10T17:11:47.222932+0000 osd.7 (osd.7) 7 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.473 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[63900]: cluster 2023-07-10T17:11:47.223137+0000 osd.2 (osd.2) 3 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.474 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[65980]: cluster 2023-07-10T17:11:46.430810+0000 mgr.y (mgr.25390) 136 : cluster 0 pgmap v70: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:48.474 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[65980]: cluster 2023-07-10T17:11:47.219428+0000 osd.7 (osd.7) 6 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.474 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[65980]: cluster 2023-07-10T17:11:47.220420+0000 mon.a (mon.0) 270 : cluster 0 osdmap e79: 8 total, 7 up, 8 in 2023-07-10T17:11:48.474 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[65980]: cluster 2023-07-10T17:11:47.220971+0000 osd.3 (osd.3) 5 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.474 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[65980]: cluster 2023-07-10T17:11:47.221704+0000 osd.4 (osd.4) 4 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.474 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[65980]: cluster 2023-07-10T17:11:47.221935+0000 osd.5 (osd.5) 5 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.475 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[65980]: cluster 2023-07-10T17:11:47.222614+0000 osd.6 (osd.6) 3 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.475 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[65980]: cluster 2023-07-10T17:11:47.222932+0000 osd.7 (osd.7) 7 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.475 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:48 smithi116 bash[65980]: cluster 2023-07-10T17:11:47.223137+0000 osd.2 (osd.2) 3 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:48 smithi192 bash[50362]: cluster 2023-07-10T17:11:46.430810+0000 mgr.y (mgr.25390) 136 : cluster 0 pgmap v70: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:11:48.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:48 smithi192 bash[50362]: cluster 2023-07-10T17:11:47.219428+0000 osd.7 (osd.7) 6 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:48 smithi192 bash[50362]: cluster 2023-07-10T17:11:47.220420+0000 mon.a (mon.0) 270 : cluster 0 osdmap e79: 8 total, 7 up, 8 in 2023-07-10T17:11:48.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:48 smithi192 bash[50362]: cluster 2023-07-10T17:11:47.220971+0000 osd.3 (osd.3) 5 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:48 smithi192 bash[50362]: cluster 2023-07-10T17:11:47.221704+0000 osd.4 (osd.4) 4 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:48 smithi192 bash[50362]: cluster 2023-07-10T17:11:47.221935+0000 osd.5 (osd.5) 5 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:48 smithi192 bash[50362]: cluster 2023-07-10T17:11:47.222614+0000 osd.6 (osd.6) 3 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:48 smithi192 bash[50362]: cluster 2023-07-10T17:11:47.222932+0000 osd.7 (osd.7) 7 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:48 smithi192 bash[50362]: cluster 2023-07-10T17:11:47.223137+0000 osd.2 (osd.2) 3 : cluster 3 failed to encode map e79 with expected crc 2023-07-10T17:11:48.799 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:48 smithi116 systemd[1]: Started Ceph osd.0 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:11:50.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:49 smithi192 bash[50362]: cluster 2023-07-10T17:11:48.432807+0000 mgr.y (mgr.25390) 137 : cluster 0 pgmap v72: 161 pgs: 24 activating+undersized, 9 active+undersized+degraded, 25 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:50.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:49 smithi192 bash[50362]: audit 2023-07-10T17:11:48.745292+0000 mon.a (mon.0) 271 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:50.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:49 smithi192 bash[50362]: audit 2023-07-10T17:11:48.753290+0000 mon.a (mon.0) 272 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:50.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:49 smithi192 bash[50362]: cluster 2023-07-10T17:11:49.214955+0000 mon.a (mon.0) 273 : cluster 3 Health check failed: Reduced data availability: 5 pgs inactive (PG_AVAILABILITY) 2023-07-10T17:11:50.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:49 smithi192 bash[50362]: cluster 2023-07-10T17:11:49.215021+0000 mon.a (mon.0) 274 : cluster 3 Health check failed: Degraded data redundancy: 116/999 objects degraded (11.612%), 9 pgs degraded (PG_DEGRADED) 2023-07-10T17:11:50.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:49 smithi116 bash[63900]: cluster 2023-07-10T17:11:48.432807+0000 mgr.y (mgr.25390) 137 : cluster 0 pgmap v72: 161 pgs: 24 activating+undersized, 9 active+undersized+degraded, 25 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:50.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:49 smithi116 bash[63900]: audit 2023-07-10T17:11:48.745292+0000 mon.a (mon.0) 271 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:50.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:49 smithi116 bash[63900]: audit 2023-07-10T17:11:48.753290+0000 mon.a (mon.0) 272 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:50.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:49 smithi116 bash[63900]: cluster 2023-07-10T17:11:49.214955+0000 mon.a (mon.0) 273 : cluster 3 Health check failed: Reduced data availability: 5 pgs inactive (PG_AVAILABILITY) 2023-07-10T17:11:50.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:49 smithi116 bash[63900]: cluster 2023-07-10T17:11:49.215021+0000 mon.a (mon.0) 274 : cluster 3 Health check failed: Degraded data redundancy: 116/999 objects degraded (11.612%), 9 pgs degraded (PG_DEGRADED) 2023-07-10T17:11:50.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:49 smithi116 bash[65980]: cluster 2023-07-10T17:11:48.432807+0000 mgr.y (mgr.25390) 137 : cluster 0 pgmap v72: 161 pgs: 24 activating+undersized, 9 active+undersized+degraded, 25 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:50.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:49 smithi116 bash[65980]: audit 2023-07-10T17:11:48.745292+0000 mon.a (mon.0) 271 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:50.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:49 smithi116 bash[65980]: audit 2023-07-10T17:11:48.753290+0000 mon.a (mon.0) 272 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:50.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:49 smithi116 bash[65980]: cluster 2023-07-10T17:11:49.214955+0000 mon.a (mon.0) 273 : cluster 3 Health check failed: Reduced data availability: 5 pgs inactive (PG_AVAILABILITY) 2023-07-10T17:11:50.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:49 smithi116 bash[65980]: cluster 2023-07-10T17:11:49.215021+0000 mon.a (mon.0) 274 : cluster 3 Health check failed: Degraded data redundancy: 116/999 objects degraded (11.612%), 9 pgs degraded (PG_DEGRADED) 2023-07-10T17:11:50.893 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:50 smithi116 bash[70193]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-07-10T17:11:50.894 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:50 smithi116 bash[70193]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2023-07-10T17:11:50.894 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:50 smithi116 bash[70193]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2023-07-10T17:11:50.894 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:50 smithi116 bash[70193]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-07-10T17:11:50.894 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:50 smithi116 bash[70193]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2023-07-10T17:11:50.894 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:50 smithi116 bash[70193]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-07-10T17:11:50.894 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:50 smithi116 bash[70193]: --> ceph-volume raw activate successful for osd ID: 0 2023-07-10T17:11:52.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:51 smithi192 bash[50362]: cluster 2023-07-10T17:11:50.433604+0000 mgr.y (mgr.25390) 138 : cluster 0 pgmap v73: 161 pgs: 24 activating+undersized, 9 active+undersized+degraded, 25 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:52.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:51 smithi116 bash[63900]: cluster 2023-07-10T17:11:50.433604+0000 mgr.y (mgr.25390) 138 : cluster 0 pgmap v73: 161 pgs: 24 activating+undersized, 9 active+undersized+degraded, 25 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:52.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:51 smithi116 bash[65980]: cluster 2023-07-10T17:11:50.433604+0000 mgr.y (mgr.25390) 138 : cluster 0 pgmap v73: 161 pgs: 24 activating+undersized, 9 active+undersized+degraded, 25 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:52.643 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:52 smithi116 bash[70480]: debug 2023-07-10T17:11:52.375+0000 7f43f5bc8700 1 -- 172.21.15.116:0/3086995899 <== mon.1 v2:172.21.15.116:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5644ecb2c4e0 con 0x5644ecb87000 2023-07-10T17:11:53.393 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:52 smithi116 bash[70480]: debug 2023-07-10T17:11:52.923+0000 7f43fed4d640 -1 Falling back to public interface 2023-07-10T17:11:53.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:53 smithi192 bash[50362]: cluster 2023-07-10T17:11:52.435544+0000 mgr.y (mgr.25390) 139 : cluster 0 pgmap v74: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:53.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:53 smithi192 bash[50362]: cluster 2023-07-10T17:11:52.748675+0000 mon.a (mon.0) 275 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive) 2023-07-10T17:11:53.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:53 smithi116 bash[65980]: cluster 2023-07-10T17:11:52.435544+0000 mgr.y (mgr.25390) 139 : cluster 0 pgmap v74: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:53.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:53 smithi116 bash[65980]: cluster 2023-07-10T17:11:52.748675+0000 mon.a (mon.0) 275 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive) 2023-07-10T17:11:53.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:53 smithi116 bash[63900]: cluster 2023-07-10T17:11:52.435544+0000 mgr.y (mgr.25390) 139 : cluster 0 pgmap v74: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:53.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:53 smithi116 bash[63900]: cluster 2023-07-10T17:11:52.748675+0000 mon.a (mon.0) 275 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive) 2023-07-10T17:11:55.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:55 smithi192 bash[50362]: cluster 2023-07-10T17:11:54.436545+0000 mgr.y (mgr.25390) 140 : cluster 0 pgmap v75: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:55.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:55 smithi116 bash[63900]: cluster 2023-07-10T17:11:54.436545+0000 mgr.y (mgr.25390) 140 : cluster 0 pgmap v75: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:55.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:55 smithi116 bash[65980]: cluster 2023-07-10T17:11:54.436545+0000 mgr.y (mgr.25390) 140 : cluster 0 pgmap v75: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:56.393 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:56 smithi116 bash[70480]: debug 2023-07-10T17:11:56.015+0000 7f43fed4d640 -1 osd.0 77 log_to_monitors true 2023-07-10T17:11:56.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:56 smithi192 bash[50362]: audit 2023-07-10T17:11:56.026456+0000 mon.c (mon.1) 14 : audit 1 from='osd.0 [v2:172.21.15.116:6802/1694270316,v1:172.21.15.116:6803/1694270316]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-10T17:11:56.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:56 smithi192 bash[50362]: audit 2023-07-10T17:11:56.027077+0000 mon.a (mon.0) 276 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-10T17:11:56.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:56 smithi192 bash[50362]: audit 2023-07-10T17:11:56.527535+0000 mon.a (mon.0) 277 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:56.893 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:11:56 smithi116 bash[70480]: debug 2023-07-10T17:11:56.615+0000 7f43ed1b4700 -1 osd.0 77 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-10T17:11:56.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:56 smithi116 bash[65980]: audit 2023-07-10T17:11:56.026456+0000 mon.c (mon.1) 14 : audit 1 from='osd.0 [v2:172.21.15.116:6802/1694270316,v1:172.21.15.116:6803/1694270316]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-10T17:11:56.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:56 smithi116 bash[65980]: audit 2023-07-10T17:11:56.027077+0000 mon.a (mon.0) 276 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-10T17:11:56.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:56 smithi116 bash[65980]: audit 2023-07-10T17:11:56.527535+0000 mon.a (mon.0) 277 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:56.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:56 smithi116 bash[63900]: audit 2023-07-10T17:11:56.026456+0000 mon.c (mon.1) 14 : audit 1 from='osd.0 [v2:172.21.15.116:6802/1694270316,v1:172.21.15.116:6803/1694270316]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-10T17:11:56.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:56 smithi116 bash[63900]: audit 2023-07-10T17:11:56.027077+0000 mon.a (mon.0) 276 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-10T17:11:56.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:56 smithi116 bash[63900]: audit 2023-07-10T17:11:56.527535+0000 mon.a (mon.0) 277 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:57.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:57 smithi192 bash[50362]: cluster 2023-07-10T17:11:56.437492+0000 mgr.y (mgr.25390) 141 : cluster 0 pgmap v76: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:57.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:57 smithi192 bash[50362]: audit 2023-07-10T17:11:56.537397+0000 mon.a (mon.0) 278 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:57.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:57 smithi192 bash[50362]: cluster 2023-07-10T17:11:56.543466+0000 osd.2 (osd.2) 4 : cluster 3 failed to encode map e80 with expected crc 2023-07-10T17:11:57.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:57 smithi192 bash[50362]: cluster 2023-07-10T17:11:56.547173+0000 osd.6 (osd.6) 4 : cluster 3 failed to encode map e80 with expected crc 2023-07-10T17:11:57.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:57 smithi192 bash[50362]: audit 2023-07-10T17:11:56.551569+0000 mon.a (mon.0) 279 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-10T17:11:57.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:57 smithi192 bash[50362]: cluster 2023-07-10T17:11:56.551644+0000 mon.a (mon.0) 280 : cluster 0 osdmap e80: 8 total, 7 up, 8 in 2023-07-10T17:11:57.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:57 smithi192 bash[50362]: audit 2023-07-10T17:11:56.554225+0000 mon.c (mon.1) 15 : audit 1 from='osd.0 [v2:172.21.15.116:6802/1694270316,v1:172.21.15.116:6803/1694270316]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:11:57.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:57 smithi192 bash[50362]: audit 2023-07-10T17:11:56.556928+0000 mon.a (mon.0) 281 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:11:57.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:57 smithi192 bash[50362]: cluster 2023-07-10T17:11:56.950126+0000 osd.7 (osd.7) 8 : cluster 3 failed to encode map e80 with expected crc 2023-07-10T17:11:57.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:57 smithi192 bash[50362]: audit 2023-07-10T17:11:57.351842+0000 mon.a (mon.0) 282 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:57.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:57 smithi192 bash[50362]: audit 2023-07-10T17:11:57.365679+0000 mon.a (mon.0) 283 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:57.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:57 smithi192 bash[50362]: cluster 2023-07-10T17:11:57.524569+0000 mon.a (mon.0) 284 : cluster 3 Health check update: Degraded data redundancy: 116/999 objects degraded (11.612%), 18 pgs degraded (PG_DEGRADED) 2023-07-10T17:11:57.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[63900]: cluster 2023-07-10T17:11:56.437492+0000 mgr.y (mgr.25390) 141 : cluster 0 pgmap v76: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:57.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[63900]: audit 2023-07-10T17:11:56.537397+0000 mon.a (mon.0) 278 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:57.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[63900]: cluster 2023-07-10T17:11:56.543466+0000 osd.2 (osd.2) 4 : cluster 3 failed to encode map e80 with expected crc 2023-07-10T17:11:57.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[63900]: cluster 2023-07-10T17:11:56.547173+0000 osd.6 (osd.6) 4 : cluster 3 failed to encode map e80 with expected crc 2023-07-10T17:11:57.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[63900]: audit 2023-07-10T17:11:56.551569+0000 mon.a (mon.0) 279 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-10T17:11:57.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[63900]: cluster 2023-07-10T17:11:56.551644+0000 mon.a (mon.0) 280 : cluster 0 osdmap e80: 8 total, 7 up, 8 in 2023-07-10T17:11:57.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[63900]: audit 2023-07-10T17:11:56.554225+0000 mon.c (mon.1) 15 : audit 1 from='osd.0 [v2:172.21.15.116:6802/1694270316,v1:172.21.15.116:6803/1694270316]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:11:57.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[63900]: audit 2023-07-10T17:11:56.556928+0000 mon.a (mon.0) 281 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:11:57.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[63900]: cluster 2023-07-10T17:11:56.950126+0000 osd.7 (osd.7) 8 : cluster 3 failed to encode map e80 with expected crc 2023-07-10T17:11:57.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[63900]: audit 2023-07-10T17:11:57.351842+0000 mon.a (mon.0) 282 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:57.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[63900]: audit 2023-07-10T17:11:57.365679+0000 mon.a (mon.0) 283 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:57.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[63900]: cluster 2023-07-10T17:11:57.524569+0000 mon.a (mon.0) 284 : cluster 3 Health check update: Degraded data redundancy: 116/999 objects degraded (11.612%), 18 pgs degraded (PG_DEGRADED) 2023-07-10T17:11:57.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[65980]: cluster 2023-07-10T17:11:56.437492+0000 mgr.y (mgr.25390) 141 : cluster 0 pgmap v76: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:57.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[65980]: audit 2023-07-10T17:11:56.537397+0000 mon.a (mon.0) 278 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:57.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[65980]: cluster 2023-07-10T17:11:56.543466+0000 osd.2 (osd.2) 4 : cluster 3 failed to encode map e80 with expected crc 2023-07-10T17:11:57.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[65980]: cluster 2023-07-10T17:11:56.547173+0000 osd.6 (osd.6) 4 : cluster 3 failed to encode map e80 with expected crc 2023-07-10T17:11:57.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[65980]: audit 2023-07-10T17:11:56.551569+0000 mon.a (mon.0) 279 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-10T17:11:57.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[65980]: cluster 2023-07-10T17:11:56.551644+0000 mon.a (mon.0) 280 : cluster 0 osdmap e80: 8 total, 7 up, 8 in 2023-07-10T17:11:57.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[65980]: audit 2023-07-10T17:11:56.554225+0000 mon.c (mon.1) 15 : audit 1 from='osd.0 [v2:172.21.15.116:6802/1694270316,v1:172.21.15.116:6803/1694270316]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:11:57.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[65980]: audit 2023-07-10T17:11:56.556928+0000 mon.a (mon.0) 281 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:11:57.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[65980]: cluster 2023-07-10T17:11:56.950126+0000 osd.7 (osd.7) 8 : cluster 3 failed to encode map e80 with expected crc 2023-07-10T17:11:57.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[65980]: audit 2023-07-10T17:11:57.351842+0000 mon.a (mon.0) 282 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:57.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[65980]: audit 2023-07-10T17:11:57.365679+0000 mon.a (mon.0) 283 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:11:57.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:57 smithi116 bash[65980]: cluster 2023-07-10T17:11:57.524569+0000 mon.a (mon.0) 284 : cluster 3 Health check update: Degraded data redundancy: 116/999 objects degraded (11.612%), 18 pgs degraded (PG_DEGRADED) 2023-07-10T17:11:58.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:58 smithi192 bash[50362]: cluster 2023-07-10T17:11:57.012618+0000 osd.3 (osd.3) 6 : cluster 3 failed to encode map e80 with expected crc 2023-07-10T17:11:58.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:58 smithi192 bash[50362]: cluster 2023-07-10T17:11:57.541619+0000 mon.a (mon.0) 285 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:11:58.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:58 smithi192 bash[50362]: cluster 2023-07-10T17:11:57.552754+0000 osd.3 (osd.3) 7 : cluster 3 failed to encode map e81 with expected crc 2023-07-10T17:11:58.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:58 smithi192 bash[50362]: cluster 2023-07-10T17:11:57.554212+0000 osd.6 (osd.6) 5 : cluster 3 failed to encode map e81 with expected crc 2023-07-10T17:11:58.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:58 smithi192 bash[50362]: cluster 2023-07-10T17:11:57.556255+0000 mon.a (mon.0) 286 : cluster 1 osd.0 [v2:172.21.15.116:6802/1694270316,v1:172.21.15.116:6803/1694270316] boot 2023-07-10T17:11:58.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:58 smithi192 bash[50362]: cluster 2023-07-10T17:11:57.556325+0000 mon.a (mon.0) 287 : cluster 0 osdmap e81: 8 total, 8 up, 8 in 2023-07-10T17:11:58.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:58 smithi192 bash[50362]: audit 2023-07-10T17:11:57.556596+0000 mon.a (mon.0) 288 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T17:11:58.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:58 smithi192 bash[50362]: cluster 2023-07-10T17:11:57.559620+0000 osd.7 (osd.7) 9 : cluster 3 failed to encode map e81 with expected crc 2023-07-10T17:11:58.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:58 smithi116 bash[65980]: cluster 2023-07-10T17:11:57.012618+0000 osd.3 (osd.3) 6 : cluster 3 failed to encode map e80 with expected crc 2023-07-10T17:11:58.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:58 smithi116 bash[65980]: cluster 2023-07-10T17:11:57.541619+0000 mon.a (mon.0) 285 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:11:58.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:58 smithi116 bash[65980]: cluster 2023-07-10T17:11:57.552754+0000 osd.3 (osd.3) 7 : cluster 3 failed to encode map e81 with expected crc 2023-07-10T17:11:58.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:58 smithi116 bash[65980]: cluster 2023-07-10T17:11:57.554212+0000 osd.6 (osd.6) 5 : cluster 3 failed to encode map e81 with expected crc 2023-07-10T17:11:58.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:58 smithi116 bash[65980]: cluster 2023-07-10T17:11:57.556255+0000 mon.a (mon.0) 286 : cluster 1 osd.0 [v2:172.21.15.116:6802/1694270316,v1:172.21.15.116:6803/1694270316] boot 2023-07-10T17:11:58.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:58 smithi116 bash[65980]: cluster 2023-07-10T17:11:57.556325+0000 mon.a (mon.0) 287 : cluster 0 osdmap e81: 8 total, 8 up, 8 in 2023-07-10T17:11:58.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:58 smithi116 bash[65980]: audit 2023-07-10T17:11:57.556596+0000 mon.a (mon.0) 288 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T17:11:58.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:58 smithi116 bash[65980]: cluster 2023-07-10T17:11:57.559620+0000 osd.7 (osd.7) 9 : cluster 3 failed to encode map e81 with expected crc 2023-07-10T17:11:58.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:58 smithi116 bash[63900]: cluster 2023-07-10T17:11:57.012618+0000 osd.3 (osd.3) 6 : cluster 3 failed to encode map e80 with expected crc 2023-07-10T17:11:58.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:58 smithi116 bash[63900]: cluster 2023-07-10T17:11:57.541619+0000 mon.a (mon.0) 285 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:11:58.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:58 smithi116 bash[63900]: cluster 2023-07-10T17:11:57.552754+0000 osd.3 (osd.3) 7 : cluster 3 failed to encode map e81 with expected crc 2023-07-10T17:11:58.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:58 smithi116 bash[63900]: cluster 2023-07-10T17:11:57.554212+0000 osd.6 (osd.6) 5 : cluster 3 failed to encode map e81 with expected crc 2023-07-10T17:11:58.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:58 smithi116 bash[63900]: cluster 2023-07-10T17:11:57.556255+0000 mon.a (mon.0) 286 : cluster 1 osd.0 [v2:172.21.15.116:6802/1694270316,v1:172.21.15.116:6803/1694270316] boot 2023-07-10T17:11:58.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:58 smithi116 bash[63900]: cluster 2023-07-10T17:11:57.556325+0000 mon.a (mon.0) 287 : cluster 0 osdmap e81: 8 total, 8 up, 8 in 2023-07-10T17:11:58.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:58 smithi116 bash[63900]: audit 2023-07-10T17:11:57.556596+0000 mon.a (mon.0) 288 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-10T17:11:58.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:58 smithi116 bash[63900]: cluster 2023-07-10T17:11:57.559620+0000 osd.7 (osd.7) 9 : cluster 3 failed to encode map e81 with expected crc 2023-07-10T17:11:59.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:59 smithi192 bash[50362]: cluster 2023-07-10T17:11:58.438499+0000 mgr.y (mgr.25390) 142 : cluster 0 pgmap v79: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:59.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:59 smithi192 bash[50362]: cluster 2023-07-10T17:11:58.562748+0000 osd.2 (osd.2) 5 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:59 smithi192 bash[50362]: cluster 2023-07-10T17:11:58.562907+0000 osd.1 (osd.1) 4 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:59 smithi192 bash[50362]: cluster 2023-07-10T17:11:58.566356+0000 mon.a (mon.0) 289 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2023-07-10T17:11:59.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:59 smithi192 bash[50362]: cluster 2023-07-10T17:11:58.566489+0000 osd.3 (osd.3) 8 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:59 smithi192 bash[50362]: cluster 2023-07-10T17:11:58.566697+0000 osd.2 (osd.2) 6 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:59 smithi192 bash[50362]: cluster 2023-07-10T17:11:58.567581+0000 osd.6 (osd.6) 6 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:59 smithi192 bash[50362]: cluster 2023-07-10T17:11:58.567792+0000 osd.7 (osd.7) 10 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:59 smithi192 bash[50362]: cluster 2023-07-10T17:11:58.567829+0000 osd.5 (osd.5) 6 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:59 smithi192 bash[50362]: cluster 2023-07-10T17:11:58.568314+0000 osd.4 (osd.4) 5 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:59 smithi192 bash[50362]: cluster 2023-07-10T17:11:58.569484+0000 osd.3 (osd.3) 9 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:59 smithi192 bash[50362]: cluster 2023-07-10T17:11:58.571227+0000 osd.6 (osd.6) 7 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:59 smithi192 bash[50362]: cluster 2023-07-10T17:11:58.571696+0000 osd.5 (osd.5) 7 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:59 smithi192 bash[50362]: cluster 2023-07-10T17:11:58.573607+0000 osd.4 (osd.4) 6 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:11:59 smithi192 bash[50362]: cluster 2023-07-10T17:11:58.573786+0000 osd.7 (osd.7) 11 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[65980]: cluster 2023-07-10T17:11:58.438499+0000 mgr.y (mgr.25390) 142 : cluster 0 pgmap v79: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:59.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[65980]: cluster 2023-07-10T17:11:58.562748+0000 osd.2 (osd.2) 5 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[65980]: cluster 2023-07-10T17:11:58.562907+0000 osd.1 (osd.1) 4 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[65980]: cluster 2023-07-10T17:11:58.566356+0000 mon.a (mon.0) 289 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2023-07-10T17:11:59.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[65980]: cluster 2023-07-10T17:11:58.566489+0000 osd.3 (osd.3) 8 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[65980]: cluster 2023-07-10T17:11:58.566697+0000 osd.2 (osd.2) 6 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[65980]: cluster 2023-07-10T17:11:58.567581+0000 osd.6 (osd.6) 6 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[65980]: cluster 2023-07-10T17:11:58.567792+0000 osd.7 (osd.7) 10 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[65980]: cluster 2023-07-10T17:11:58.567829+0000 osd.5 (osd.5) 6 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[65980]: cluster 2023-07-10T17:11:58.568314+0000 osd.4 (osd.4) 5 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[65980]: cluster 2023-07-10T17:11:58.569484+0000 osd.3 (osd.3) 9 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[65980]: cluster 2023-07-10T17:11:58.571227+0000 osd.6 (osd.6) 7 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[65980]: cluster 2023-07-10T17:11:58.571696+0000 osd.5 (osd.5) 7 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[65980]: cluster 2023-07-10T17:11:58.573607+0000 osd.4 (osd.4) 6 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[65980]: cluster 2023-07-10T17:11:58.573786+0000 osd.7 (osd.7) 11 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[63900]: cluster 2023-07-10T17:11:58.438499+0000 mgr.y (mgr.25390) 142 : cluster 0 pgmap v79: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:11:59.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[63900]: cluster 2023-07-10T17:11:58.562748+0000 osd.2 (osd.2) 5 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[63900]: cluster 2023-07-10T17:11:58.562907+0000 osd.1 (osd.1) 4 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[63900]: cluster 2023-07-10T17:11:58.566356+0000 mon.a (mon.0) 289 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2023-07-10T17:11:59.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[63900]: cluster 2023-07-10T17:11:58.566489+0000 osd.3 (osd.3) 8 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[63900]: cluster 2023-07-10T17:11:58.566697+0000 osd.2 (osd.2) 6 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[63900]: cluster 2023-07-10T17:11:58.567581+0000 osd.6 (osd.6) 6 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[63900]: cluster 2023-07-10T17:11:58.567792+0000 osd.7 (osd.7) 10 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[63900]: cluster 2023-07-10T17:11:58.567829+0000 osd.5 (osd.5) 6 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[63900]: cluster 2023-07-10T17:11:58.568314+0000 osd.4 (osd.4) 5 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[63900]: cluster 2023-07-10T17:11:58.569484+0000 osd.3 (osd.3) 9 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[63900]: cluster 2023-07-10T17:11:58.571227+0000 osd.6 (osd.6) 7 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[63900]: cluster 2023-07-10T17:11:58.571696+0000 osd.5 (osd.5) 7 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[63900]: cluster 2023-07-10T17:11:58.573607+0000 osd.4 (osd.4) 6 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:11:59.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:11:59 smithi116 bash[63900]: cluster 2023-07-10T17:11:58.573786+0000 osd.7 (osd.7) 11 : cluster 3 failed to encode map e82 with expected crc 2023-07-10T17:12:01.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:01 smithi192 bash[50362]: cluster 2023-07-10T17:12:00.439515+0000 mgr.y (mgr.25390) 143 : cluster 0 pgmap v81: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:12:01.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:01 smithi116 bash[63900]: cluster 2023-07-10T17:12:00.439515+0000 mgr.y (mgr.25390) 143 : cluster 0 pgmap v81: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:12:01.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:01 smithi116 bash[65980]: cluster 2023-07-10T17:12:00.439515+0000 mgr.y (mgr.25390) 143 : cluster 0 pgmap v81: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 116/999 objects degraded (11.612%) 2023-07-10T17:12:03.798 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:03 smithi116 bash[63900]: cluster 2023-07-10T17:12:02.441172+0000 mgr.y (mgr.25390) 144 : cluster 0 pgmap v82: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:03.798 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:03 smithi116 bash[63900]: cluster 2023-07-10T17:12:02.568600+0000 mon.a (mon.0) 290 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 116/999 objects degraded (11.612%), 18 pgs degraded) 2023-07-10T17:12:03.799 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:03 smithi116 bash[65980]: cluster 2023-07-10T17:12:02.441172+0000 mgr.y (mgr.25390) 144 : cluster 0 pgmap v82: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:03.799 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:03 smithi116 bash[65980]: cluster 2023-07-10T17:12:02.568600+0000 mon.a (mon.0) 290 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 116/999 objects degraded (11.612%), 18 pgs degraded) 2023-07-10T17:12:03.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:03 smithi192 bash[50362]: cluster 2023-07-10T17:12:02.441172+0000 mgr.y (mgr.25390) 144 : cluster 0 pgmap v82: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:03.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:03 smithi192 bash[50362]: cluster 2023-07-10T17:12:02.568600+0000 mon.a (mon.0) 290 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 116/999 objects degraded (11.612%), 18 pgs degraded) 2023-07-10T17:12:05.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:05 smithi116 bash[63900]: cluster 2023-07-10T17:12:04.441838+0000 mgr.y (mgr.25390) 145 : cluster 0 pgmap v83: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:05.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:05 smithi116 bash[65980]: cluster 2023-07-10T17:12:04.441838+0000 mgr.y (mgr.25390) 145 : cluster 0 pgmap v83: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:05.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:05 smithi192 bash[50362]: cluster 2023-07-10T17:12:04.441838+0000 mgr.y (mgr.25390) 145 : cluster 0 pgmap v83: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:06.574 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:12:07.320 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:12:07.320 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (5m) 10s ago 11m 10.7M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:12:07.321 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (3m) 81s ago 11m 25.6M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:12:07.321 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 error 10s ago 9m - - 2023-07-10T17:12:07.321 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (3m) 81s ago 15m 428M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:12:07.321 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (3m) 10s ago 17m 471M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:12:07.321 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (2m) 10s ago 17m 55.8M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:12:07.321 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (95s) 81s ago 16m 40.2M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 74a082a2e795 2023-07-10T17:12:07.321 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (2m) 10s ago 16m 48.2M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:12:07.322 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (5m) 10s ago 11m 9408k - 0.18.1 e5a616e4b9cf 3bd058d91353 2023-07-10T17:12:07.322 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (4m) 81s ago 11m 3607k - 0.18.1 e5a616e4b9cf 63551615c679 2023-07-10T17:12:07.322 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (15s) 10s ago 15m - 4096M 18.1.2-30-g8507faa7 17e8f0d79247 0562f587fbca 2023-07-10T17:12:07.322 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (15m) 10s ago 15m 54.8M 4096M 16.2.0 24ecd6d5f14c c157509b3fd4 2023-07-10T17:12:07.322 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (14m) 10s ago 14m 50.5M 4096M 16.2.0 24ecd6d5f14c f1fd77b3c454 2023-07-10T17:12:07.322 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (14m) 10s ago 14m 56.1M 4096M 16.2.0 24ecd6d5f14c 60949d479c11 2023-07-10T17:12:07.322 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (13m) 81s ago 13m 55.1M 4096M 16.2.0 24ecd6d5f14c 75fc0b36aab8 2023-07-10T17:12:07.322 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (13m) 81s ago 13m 52.8M 4096M 16.2.0 24ecd6d5f14c d673065333dd 2023-07-10T17:12:07.323 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (12m) 81s ago 13m 53.7M 4096M 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:12:07.323 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (12m) 81s ago 12m 56.7M 4096M 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:12:07.323 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 error 81s ago 11m - - 2023-07-10T17:12:07.323 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (9m) 10s ago 9m 55.3M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:12:07.323 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (9m) 81s ago 9m 54.8M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:12:07.323 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (9m) 10s ago 9m 55.0M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:12:07.323 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (9m) 81s ago 9m 56.6M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:12:07.761 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:12:07.761 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:12:07.761 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:12:07.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:12:07.762 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:12:07.762 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:12:07.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:12:07.762 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:12:07.762 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 7, 2023-07-10T17:12:07.762 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 1 2023-07-10T17:12:07.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:12:07.762 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:12:07.763 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:12:07.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:12:07.763 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:12:07.763 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 11, 2023-07-10T17:12:07.763 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 6 2023-07-10T17:12:07.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:12:07.763 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:12:07.870 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:07 smithi116 bash[65980]: cluster 2023-07-10T17:12:06.442358+0000 mgr.y (mgr.25390) 146 : cluster 0 pgmap v84: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:07.871 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:07 smithi116 bash[65980]: audit 2023-07-10T17:12:06.555309+0000 mgr.y (mgr.25390) 147 : audit 0 from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:07.871 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:07 smithi116 bash[65980]: audit 2023-07-10T17:12:06.928431+0000 mgr.y (mgr.25390) 148 : audit 0 from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:07.871 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:07 smithi116 bash[65980]: audit 2023-07-10T17:12:07.462624+0000 mon.a (mon.0) 291 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:07.871 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:07 smithi116 bash[65980]: audit 2023-07-10T17:12:07.474345+0000 mon.a (mon.0) 292 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:07.871 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:07 smithi116 bash[65980]: audit 2023-07-10T17:12:07.476012+0000 mon.a (mon.0) 293 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:12:07.871 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:07 smithi116 bash[65980]: audit 2023-07-10T17:12:07.477105+0000 mon.a (mon.0) 294 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:12:07.871 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:07 smithi116 bash[65980]: audit 2023-07-10T17:12:07.485378+0000 mon.a (mon.0) 295 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:07.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:07 smithi116 bash[63900]: cluster 2023-07-10T17:12:06.442358+0000 mgr.y (mgr.25390) 146 : cluster 0 pgmap v84: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:07.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:07 smithi116 bash[63900]: audit 2023-07-10T17:12:06.555309+0000 mgr.y (mgr.25390) 147 : audit 0 from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:07.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:07 smithi116 bash[63900]: audit 2023-07-10T17:12:06.928431+0000 mgr.y (mgr.25390) 148 : audit 0 from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:07.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:07 smithi116 bash[63900]: audit 2023-07-10T17:12:07.462624+0000 mon.a (mon.0) 291 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:07.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:07 smithi116 bash[63900]: audit 2023-07-10T17:12:07.474345+0000 mon.a (mon.0) 292 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:07.873 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:07 smithi116 bash[63900]: audit 2023-07-10T17:12:07.476012+0000 mon.a (mon.0) 293 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:12:07.873 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:07 smithi116 bash[63900]: audit 2023-07-10T17:12:07.477105+0000 mon.a (mon.0) 294 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:12:07.873 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:07 smithi116 bash[63900]: audit 2023-07-10T17:12:07.485378+0000 mon.a (mon.0) 295 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:07.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:07 smithi192 bash[50362]: cluster 2023-07-10T17:12:06.442358+0000 mgr.y (mgr.25390) 146 : cluster 0 pgmap v84: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:07.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:07 smithi192 bash[50362]: audit 2023-07-10T17:12:06.555309+0000 mgr.y (mgr.25390) 147 : audit 0 from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:07.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:07 smithi192 bash[50362]: audit 2023-07-10T17:12:06.928431+0000 mgr.y (mgr.25390) 148 : audit 0 from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:07.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:07 smithi192 bash[50362]: audit 2023-07-10T17:12:07.462624+0000 mon.a (mon.0) 291 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:07.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:07 smithi192 bash[50362]: audit 2023-07-10T17:12:07.474345+0000 mon.a (mon.0) 292 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:07.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:07 smithi192 bash[50362]: audit 2023-07-10T17:12:07.476012+0000 mon.a (mon.0) 293 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:12:07.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:07 smithi192 bash[50362]: audit 2023-07-10T17:12:07.477105+0000 mon.a (mon.0) 294 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:12:07.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:07 smithi192 bash[50362]: audit 2023-07-10T17:12:07.485378+0000 mon.a (mon.0) 295 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:08.127 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:12:08.127 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:12:08.128 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:12:08.128 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:12:08.128 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:12:08.129 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2023-07-10T17:12:08.129 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2023-07-10T17:12:08.129 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:12:08.130 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "6/23 daemons upgraded", 2023-07-10T17:12:08.130 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading osd daemons", 2023-07-10T17:12:08.130 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:12:08.130 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:12:08.561 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:12:08.562 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:12:08.562 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:12:08.562 INFO:teuthology.orchestra.run.smithi116.stdout: daemon prometheus.a on smithi192 is in error state 2023-07-10T17:12:08.860 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[65980]: audit 2023-07-10T17:12:07.300727+0000 mgr.y (mgr.25390) 149 : audit 0 from='client.34467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:08.860 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[65980]: cephadm 2023-07-10T17:12:07.477948+0000 mgr.y (mgr.25390) 150 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:12:08.861 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[65980]: cluster 2023-07-10T17:12:07.479413+0000 mgr.y (mgr.25390) 151 : cluster 0 pgmap v85: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:08.861 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[65980]: cluster 2023-07-10T17:12:07.479824+0000 mgr.y (mgr.25390) 152 : cluster 0 pgmap v86: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:08.861 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[65980]: audit 2023-07-10T17:12:07.566403+0000 mon.a (mon.0) 296 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:12:08.861 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[65980]: audit 2023-07-10T17:12:07.569788+0000 mon.a (mon.0) 297 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:08.861 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[65980]: audit 2023-07-10T17:12:07.571765+0000 mon.a (mon.0) 298 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:08.862 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[65980]: audit 2023-07-10T17:12:07.573257+0000 mon.a (mon.0) 299 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:08.862 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[65980]: audit 2023-07-10T17:12:07.575059+0000 mon.a (mon.0) 300 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-07-10T17:12:08.862 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[65980]: audit 2023-07-10T17:12:07.575292+0000 mgr.y (mgr.25390) 153 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-07-10T17:12:08.863 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[65980]: cephadm 2023-07-10T17:12:07.576491+0000 mgr.y (mgr.25390) 154 : cephadm 1 Upgrade: osd.1 is safe to restart 2023-07-10T17:12:08.863 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[65980]: audit 2023-07-10T17:12:07.756744+0000 mon.a (mon.0) 301 : audit 0 from='client.34473 172.21.15.116:0/2917076378' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:08.863 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[65980]: audit 2023-07-10T17:12:08.122513+0000 mgr.y (mgr.25390) 155 : audit 0 from='client.44335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:08.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[63900]: audit 2023-07-10T17:12:07.300727+0000 mgr.y (mgr.25390) 149 : audit 0 from='client.34467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:08.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[63900]: cephadm 2023-07-10T17:12:07.477948+0000 mgr.y (mgr.25390) 150 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:12:08.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[63900]: cluster 2023-07-10T17:12:07.479413+0000 mgr.y (mgr.25390) 151 : cluster 0 pgmap v85: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:08.865 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[63900]: cluster 2023-07-10T17:12:07.479824+0000 mgr.y (mgr.25390) 152 : cluster 0 pgmap v86: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:08.865 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[63900]: audit 2023-07-10T17:12:07.566403+0000 mon.a (mon.0) 296 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:12:08.865 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[63900]: audit 2023-07-10T17:12:07.569788+0000 mon.a (mon.0) 297 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:08.865 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[63900]: audit 2023-07-10T17:12:07.571765+0000 mon.a (mon.0) 298 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:08.866 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[63900]: audit 2023-07-10T17:12:07.573257+0000 mon.a (mon.0) 299 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:08.866 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[63900]: audit 2023-07-10T17:12:07.575059+0000 mon.a (mon.0) 300 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-07-10T17:12:08.866 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[63900]: audit 2023-07-10T17:12:07.575292+0000 mgr.y (mgr.25390) 153 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-07-10T17:12:08.866 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[63900]: cephadm 2023-07-10T17:12:07.576491+0000 mgr.y (mgr.25390) 154 : cephadm 1 Upgrade: osd.1 is safe to restart 2023-07-10T17:12:08.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[63900]: audit 2023-07-10T17:12:07.756744+0000 mon.a (mon.0) 301 : audit 0 from='client.34473 172.21.15.116:0/2917076378' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:08.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:08 smithi116 bash[63900]: audit 2023-07-10T17:12:08.122513+0000 mgr.y (mgr.25390) 155 : audit 0 from='client.44335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:08.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:08 smithi192 bash[50362]: audit 2023-07-10T17:12:07.300727+0000 mgr.y (mgr.25390) 149 : audit 0 from='client.34467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:08.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:08 smithi192 bash[50362]: cephadm 2023-07-10T17:12:07.477948+0000 mgr.y (mgr.25390) 150 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:12:08.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:08 smithi192 bash[50362]: cluster 2023-07-10T17:12:07.479413+0000 mgr.y (mgr.25390) 151 : cluster 0 pgmap v85: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:08.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:08 smithi192 bash[50362]: cluster 2023-07-10T17:12:07.479824+0000 mgr.y (mgr.25390) 152 : cluster 0 pgmap v86: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:08.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:08 smithi192 bash[50362]: audit 2023-07-10T17:12:07.566403+0000 mon.a (mon.0) 296 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:12:08.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:08 smithi192 bash[50362]: audit 2023-07-10T17:12:07.569788+0000 mon.a (mon.0) 297 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:08.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:08 smithi192 bash[50362]: audit 2023-07-10T17:12:07.571765+0000 mon.a (mon.0) 298 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:08.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:08 smithi192 bash[50362]: audit 2023-07-10T17:12:07.573257+0000 mon.a (mon.0) 299 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:08.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:08 smithi192 bash[50362]: audit 2023-07-10T17:12:07.575059+0000 mon.a (mon.0) 300 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-07-10T17:12:08.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:08 smithi192 bash[50362]: audit 2023-07-10T17:12:07.575292+0000 mgr.y (mgr.25390) 153 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-07-10T17:12:08.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:08 smithi192 bash[50362]: cephadm 2023-07-10T17:12:07.576491+0000 mgr.y (mgr.25390) 154 : cephadm 1 Upgrade: osd.1 is safe to restart 2023-07-10T17:12:08.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:08 smithi192 bash[50362]: audit 2023-07-10T17:12:07.756744+0000 mon.a (mon.0) 301 : audit 0 from='client.34473 172.21.15.116:0/2917076378' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:08.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:08 smithi192 bash[50362]: audit 2023-07-10T17:12:08.122513+0000 mgr.y (mgr.25390) 155 : audit 0 from='client.44335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:09.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:09 smithi192 bash[50362]: audit 2023-07-10T17:12:08.557770+0000 mon.a (mon.0) 302 : audit 0 from='client.44341 172.21.15.116:0/1725736283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:12:09.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:09 smithi116 bash[63900]: audit 2023-07-10T17:12:08.557770+0000 mon.a (mon.0) 302 : audit 0 from='client.44341 172.21.15.116:0/1725736283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:12:09.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:09 smithi116 bash[65980]: audit 2023-07-10T17:12:08.557770+0000 mon.a (mon.0) 302 : audit 0 from='client.44341 172.21.15.116:0/1725736283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:12:11.119 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:10 smithi116 bash[65980]: cluster 2023-07-10T17:12:09.480498+0000 mgr.y (mgr.25390) 156 : cluster 0 pgmap v87: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:11.119 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:10 smithi116 bash[65980]: cephadm 2023-07-10T17:12:09.846562+0000 mgr.y (mgr.25390) 157 : cephadm 1 Upgrade: Updating osd.1 2023-07-10T17:12:11.119 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:10 smithi116 bash[65980]: audit 2023-07-10T17:12:09.855790+0000 mon.a (mon.0) 303 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:11.119 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:10 smithi116 bash[65980]: audit 2023-07-10T17:12:09.857694+0000 mon.a (mon.0) 304 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-10T17:12:11.120 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:10 smithi116 bash[65980]: audit 2023-07-10T17:12:09.859065+0000 mon.a (mon.0) 305 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:12:11.120 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:10 smithi116 bash[65980]: cephadm 2023-07-10T17:12:09.861977+0000 mgr.y (mgr.25390) 158 : cephadm 1 Deploying daemon osd.1 on smithi116 2023-07-10T17:12:11.120 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:10 smithi116 bash[63900]: cluster 2023-07-10T17:12:09.480498+0000 mgr.y (mgr.25390) 156 : cluster 0 pgmap v87: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:11.120 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:10 smithi116 bash[63900]: cephadm 2023-07-10T17:12:09.846562+0000 mgr.y (mgr.25390) 157 : cephadm 1 Upgrade: Updating osd.1 2023-07-10T17:12:11.120 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:10 smithi116 bash[63900]: audit 2023-07-10T17:12:09.855790+0000 mon.a (mon.0) 303 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:11.121 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:10 smithi116 bash[63900]: audit 2023-07-10T17:12:09.857694+0000 mon.a (mon.0) 304 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-10T17:12:11.121 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:10 smithi116 bash[63900]: audit 2023-07-10T17:12:09.859065+0000 mon.a (mon.0) 305 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:12:11.121 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:10 smithi116 bash[63900]: cephadm 2023-07-10T17:12:09.861977+0000 mgr.y (mgr.25390) 158 : cephadm 1 Deploying daemon osd.1 on smithi116 2023-07-10T17:12:11.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:10 smithi192 bash[50362]: cluster 2023-07-10T17:12:09.480498+0000 mgr.y (mgr.25390) 156 : cluster 0 pgmap v87: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:11.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:10 smithi192 bash[50362]: cephadm 2023-07-10T17:12:09.846562+0000 mgr.y (mgr.25390) 157 : cephadm 1 Upgrade: Updating osd.1 2023-07-10T17:12:11.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:10 smithi192 bash[50362]: audit 2023-07-10T17:12:09.855790+0000 mon.a (mon.0) 303 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:11.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:10 smithi192 bash[50362]: audit 2023-07-10T17:12:09.857694+0000 mon.a (mon.0) 304 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-10T17:12:11.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:10 smithi192 bash[50362]: audit 2023-07-10T17:12:09.859065+0000 mon.a (mon.0) 305 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:12:11.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:10 smithi192 bash[50362]: cephadm 2023-07-10T17:12:09.861977+0000 mgr.y (mgr.25390) 158 : cephadm 1 Deploying daemon osd.1 on smithi116 2023-07-10T17:12:12.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:12 smithi192 bash[50362]: audit 2023-07-10T17:12:11.376929+0000 mon.a (mon.0) 306 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:12.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:12 smithi192 bash[50362]: cluster 2023-07-10T17:12:11.481224+0000 mgr.y (mgr.25390) 159 : cluster 0 pgmap v88: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:12.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:12 smithi116 bash[63900]: audit 2023-07-10T17:12:11.376929+0000 mon.a (mon.0) 306 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:12.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:12 smithi116 bash[63900]: cluster 2023-07-10T17:12:11.481224+0000 mgr.y (mgr.25390) 159 : cluster 0 pgmap v88: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:12.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:12 smithi116 bash[65980]: audit 2023-07-10T17:12:11.376929+0000 mon.a (mon.0) 306 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:12.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:12 smithi116 bash[65980]: cluster 2023-07-10T17:12:11.481224+0000 mgr.y (mgr.25390) 159 : cluster 0 pgmap v88: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:14.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:14 smithi116 bash[63900]: cluster 2023-07-10T17:12:13.482325+0000 mgr.y (mgr.25390) 160 : cluster 0 pgmap v89: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:14.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:14 smithi116 bash[65980]: cluster 2023-07-10T17:12:13.482325+0000 mgr.y (mgr.25390) 160 : cluster 0 pgmap v89: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:14.644 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:14 smithi116 systemd[1]: Stopping Ceph osd.1 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:12:14.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:14 smithi192 bash[50362]: cluster 2023-07-10T17:12:13.482325+0000 mgr.y (mgr.25390) 160 : cluster 0 pgmap v89: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:15.143 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:14 smithi116 bash[33698]: debug 2023-07-10T17:12:14.687+0000 7f9dfd4d5700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:12:15.144 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:14 smithi116 bash[33698]: debug 2023-07-10T17:12:14.687+0000 7f9dfd4d5700 -1 osd.1 82 *** Got signal Terminated *** 2023-07-10T17:12:15.144 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:14 smithi116 bash[33698]: debug 2023-07-10T17:12:14.687+0000 7f9dfd4d5700 -1 osd.1 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-10T17:12:15.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.753496+0000 mon.a (mon.0) 307 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.753566+0000 mon.a (mon.0) 308 : cluster 1 osd.1 failed (root=default,host=smithi116) (connection refused reported by osd.0) 2023-07-10T17:12:15.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.753756+0000 mon.a (mon.0) 309 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.955886+0000 mon.a (mon.0) 310 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-10T17:12:15.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.956240+0000 mon.a (mon.0) 311 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.956364+0000 mon.a (mon.0) 312 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.956833+0000 mon.a (mon.0) 313 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.957118+0000 mon.a (mon.0) 314 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-10T17:12:15.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.957392+0000 mon.a (mon.0) 315 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-10T17:12:15.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.957687+0000 mon.a (mon.0) 316 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.958084+0000 mon.a (mon.0) 317 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.958375+0000 mon.a (mon.0) 318 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.958651+0000 mon.a (mon.0) 319 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.958942+0000 mon.a (mon.0) 320 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-10T17:12:15.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.959239+0000 mon.a (mon.0) 321 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.959524+0000 mon.a (mon.0) 322 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.959694+0000 mon.a (mon.0) 323 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.959882+0000 mon.a (mon.0) 324 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.960107+0000 mon.a (mon.0) 325 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.960338+0000 mon.a (mon.0) 326 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:14.960577+0000 mon.a (mon.0) 327 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.359711+0000 mon.a (mon.0) 328 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-10T17:12:15.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.360205+0000 mon.a (mon.0) 329 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.360563+0000 mon.a (mon.0) 330 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.360833+0000 mon.a (mon.0) 331 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.360947+0000 mon.a (mon.0) 332 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-10T17:12:15.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.361268+0000 mon.a (mon.0) 333 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-10T17:12:15.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.361384+0000 mon.a (mon.0) 334 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.361733+0000 mon.a (mon.0) 335 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.892 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.361874+0000 mon.a (mon.0) 336 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.892 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.362240+0000 mon.a (mon.0) 337 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.892 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.362352+0000 mon.a (mon.0) 338 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-10T17:12:15.892 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.362641+0000 mon.a (mon.0) 339 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.892 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.362839+0000 mon.a (mon.0) 340 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.892 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.363263+0000 mon.a (mon.0) 341 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.892 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.363484+0000 mon.a (mon.0) 342 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.893 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.363756+0000 mon.a (mon.0) 343 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.893 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.364149+0000 mon.a (mon.0) 344 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.893 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.364491+0000 mon.a (mon.0) 345 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.893 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:15 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.533463+0000 mon.a (mon.0) 346 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-10T17:12:15.894 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[72287]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-osd.1 2023-07-10T17:12:15.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.753496+0000 mon.a (mon.0) 307 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.753566+0000 mon.a (mon.0) 308 : cluster 1 osd.1 failed (root=default,host=smithi116) (connection refused reported by osd.0) 2023-07-10T17:12:15.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.753756+0000 mon.a (mon.0) 309 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.955886+0000 mon.a (mon.0) 310 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-10T17:12:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.956240+0000 mon.a (mon.0) 311 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.956364+0000 mon.a (mon.0) 312 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.956833+0000 mon.a (mon.0) 313 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.957118+0000 mon.a (mon.0) 314 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-10T17:12:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.957392+0000 mon.a (mon.0) 315 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-10T17:12:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.957687+0000 mon.a (mon.0) 316 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.958084+0000 mon.a (mon.0) 317 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.958375+0000 mon.a (mon.0) 318 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.958651+0000 mon.a (mon.0) 319 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.958942+0000 mon.a (mon.0) 320 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-10T17:12:15.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.959239+0000 mon.a (mon.0) 321 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.959524+0000 mon.a (mon.0) 322 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.959694+0000 mon.a (mon.0) 323 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.959882+0000 mon.a (mon.0) 324 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.960107+0000 mon.a (mon.0) 325 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.960338+0000 mon.a (mon.0) 326 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:14.960577+0000 mon.a (mon.0) 327 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.359711+0000 mon.a (mon.0) 328 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-10T17:12:15.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.360205+0000 mon.a (mon.0) 329 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.360563+0000 mon.a (mon.0) 330 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.360833+0000 mon.a (mon.0) 331 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.360947+0000 mon.a (mon.0) 332 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-10T17:12:15.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.753496+0000 mon.a (mon.0) 307 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.753566+0000 mon.a (mon.0) 308 : cluster 1 osd.1 failed (root=default,host=smithi116) (connection refused reported by osd.0) 2023-07-10T17:12:15.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.753756+0000 mon.a (mon.0) 309 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.955886+0000 mon.a (mon.0) 310 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-10T17:12:15.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.956240+0000 mon.a (mon.0) 311 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.956364+0000 mon.a (mon.0) 312 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.956833+0000 mon.a (mon.0) 313 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.957118+0000 mon.a (mon.0) 314 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-10T17:12:15.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.957392+0000 mon.a (mon.0) 315 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-10T17:12:15.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.957687+0000 mon.a (mon.0) 316 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.958084+0000 mon.a (mon.0) 317 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.958375+0000 mon.a (mon.0) 318 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.958651+0000 mon.a (mon.0) 319 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.958942+0000 mon.a (mon.0) 320 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-10T17:12:15.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.959239+0000 mon.a (mon.0) 321 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.959524+0000 mon.a (mon.0) 322 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.959694+0000 mon.a (mon.0) 323 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.959882+0000 mon.a (mon.0) 324 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.960107+0000 mon.a (mon.0) 325 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.960338+0000 mon.a (mon.0) 326 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:14.960577+0000 mon.a (mon.0) 327 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.359711+0000 mon.a (mon.0) 328 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-10T17:12:15.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.360205+0000 mon.a (mon.0) 329 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.903 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.360563+0000 mon.a (mon.0) 330 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.903 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.360833+0000 mon.a (mon.0) 331 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.903 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.360947+0000 mon.a (mon.0) 332 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-10T17:12:15.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.361268+0000 mon.a (mon.0) 333 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-10T17:12:15.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.361384+0000 mon.a (mon.0) 334 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.361733+0000 mon.a (mon.0) 335 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.361874+0000 mon.a (mon.0) 336 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.362240+0000 mon.a (mon.0) 337 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.362352+0000 mon.a (mon.0) 338 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-10T17:12:15.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.362641+0000 mon.a (mon.0) 339 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.362839+0000 mon.a (mon.0) 340 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.363263+0000 mon.a (mon.0) 341 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.363484+0000 mon.a (mon.0) 342 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.363756+0000 mon.a (mon.0) 343 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.364149+0000 mon.a (mon.0) 344 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.364491+0000 mon.a (mon.0) 345 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.533463+0000 mon.a (mon.0) 346 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-10T17:12:15.906 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.361268+0000 mon.a (mon.0) 333 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-10T17:12:15.906 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.361384+0000 mon.a (mon.0) 334 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.906 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.361733+0000 mon.a (mon.0) 335 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.906 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.361874+0000 mon.a (mon.0) 336 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.906 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.362240+0000 mon.a (mon.0) 337 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.906 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.362352+0000 mon.a (mon.0) 338 : cluster 0 osd.1 reported immediately failed by osd.7 2023-07-10T17:12:15.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.362641+0000 mon.a (mon.0) 339 : cluster 0 osd.1 reported immediately failed by osd.4 2023-07-10T17:12:15.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.362839+0000 mon.a (mon.0) 340 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.363263+0000 mon.a (mon.0) 341 : cluster 0 osd.1 reported immediately failed by osd.2 2023-07-10T17:12:15.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.363484+0000 mon.a (mon.0) 342 : cluster 0 osd.1 reported immediately failed by osd.3 2023-07-10T17:12:15.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.363756+0000 mon.a (mon.0) 343 : cluster 0 osd.1 reported immediately failed by osd.6 2023-07-10T17:12:15.908 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.364149+0000 mon.a (mon.0) 344 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.908 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.364491+0000 mon.a (mon.0) 345 : cluster 0 osd.1 reported immediately failed by osd.0 2023-07-10T17:12:15.908 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:15 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.533463+0000 mon.a (mon.0) 346 : cluster 0 osd.1 reported immediately failed by osd.5 2023-07-10T17:12:16.851 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.483043+0000 mgr.y (mgr.25390) 161 : cluster 0 pgmap v90: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:16.851 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.538411+0000 mon.a (mon.0) 347 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:12:16.852 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.547901+0000 osd.7 (osd.7) 12 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.852 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.549026+0000 osd.6 (osd.6) 8 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.852 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.557959+0000 osd.6 (osd.6) 9 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.852 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.559600+0000 mon.a (mon.0) 348 : cluster 0 osdmap e83: 8 total, 7 up, 8 in 2023-07-10T17:12:16.852 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.560783+0000 osd.3 (osd.3) 10 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.852 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.561016+0000 osd.5 (osd.5) 8 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.852 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[65980]: cluster 2023-07-10T17:12:15.562053+0000 osd.4 (osd.4) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.853 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.483043+0000 mgr.y (mgr.25390) 161 : cluster 0 pgmap v90: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:16.853 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.538411+0000 mon.a (mon.0) 347 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:12:16.853 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.547901+0000 osd.7 (osd.7) 12 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.853 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.549026+0000 osd.6 (osd.6) 8 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.853 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.557959+0000 osd.6 (osd.6) 9 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.853 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.559600+0000 mon.a (mon.0) 348 : cluster 0 osdmap e83: 8 total, 7 up, 8 in 2023-07-10T17:12:16.854 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.560783+0000 osd.3 (osd.3) 10 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.854 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.561016+0000 osd.5 (osd.5) 8 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.854 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:16 smithi116 bash[63900]: cluster 2023-07-10T17:12:15.562053+0000 osd.4 (osd.4) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:16 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.483043+0000 mgr.y (mgr.25390) 161 : cluster 0 pgmap v90: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:16.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:16 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.538411+0000 mon.a (mon.0) 347 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:12:16.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:16 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.547901+0000 osd.7 (osd.7) 12 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:16 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.549026+0000 osd.6 (osd.6) 8 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:16 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.557959+0000 osd.6 (osd.6) 9 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:16 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.559600+0000 mon.a (mon.0) 348 : cluster 0 osdmap e83: 8 total, 7 up, 8 in 2023-07-10T17:12:16.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:16 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.560783+0000 osd.3 (osd.3) 10 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:16 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.561016+0000 osd.5 (osd.5) 8 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:16.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:16 smithi192 bash[50362]: cluster 2023-07-10T17:12:15.562053+0000 osd.4 (osd.4) 7 : cluster 3 failed to encode map e83 with expected crc 2023-07-10T17:12:17.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[63900]: cluster 2023-07-10T17:12:16.557320+0000 osd.2 (osd.2) 7 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[63900]: cluster 2023-07-10T17:12:16.558886+0000 osd.6 (osd.6) 10 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[63900]: cluster 2023-07-10T17:12:16.559940+0000 mon.a (mon.0) 349 : cluster 0 osdmap e84: 8 total, 7 up, 8 in 2023-07-10T17:12:17.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[63900]: cluster 2023-07-10T17:12:16.560163+0000 osd.2 (osd.2) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[63900]: cluster 2023-07-10T17:12:16.560352+0000 osd.3 (osd.3) 11 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[63900]: cluster 2023-07-10T17:12:16.561232+0000 osd.4 (osd.4) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[63900]: cluster 2023-07-10T17:12:16.561290+0000 osd.5 (osd.5) 9 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[63900]: cluster 2023-07-10T17:12:16.561397+0000 osd.7 (osd.7) 13 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[63900]: cluster 2023-07-10T17:12:16.562998+0000 osd.6 (osd.6) 11 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[65980]: cluster 2023-07-10T17:12:16.557320+0000 osd.2 (osd.2) 7 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[65980]: cluster 2023-07-10T17:12:16.558886+0000 osd.6 (osd.6) 10 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[65980]: cluster 2023-07-10T17:12:16.559940+0000 mon.a (mon.0) 349 : cluster 0 osdmap e84: 8 total, 7 up, 8 in 2023-07-10T17:12:17.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[65980]: cluster 2023-07-10T17:12:16.560163+0000 osd.2 (osd.2) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[65980]: cluster 2023-07-10T17:12:16.560352+0000 osd.3 (osd.3) 11 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[65980]: cluster 2023-07-10T17:12:16.561232+0000 osd.4 (osd.4) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[65980]: cluster 2023-07-10T17:12:16.561290+0000 osd.5 (osd.5) 9 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[65980]: cluster 2023-07-10T17:12:16.561397+0000 osd.7 (osd.7) 13 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:17 smithi116 bash[65980]: cluster 2023-07-10T17:12:16.562998+0000 osd.6 (osd.6) 11 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:17 smithi192 bash[50362]: cluster 2023-07-10T17:12:16.557320+0000 osd.2 (osd.2) 7 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:17 smithi192 bash[50362]: cluster 2023-07-10T17:12:16.558886+0000 osd.6 (osd.6) 10 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:17 smithi192 bash[50362]: cluster 2023-07-10T17:12:16.559940+0000 mon.a (mon.0) 349 : cluster 0 osdmap e84: 8 total, 7 up, 8 in 2023-07-10T17:12:17.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:17 smithi192 bash[50362]: cluster 2023-07-10T17:12:16.560163+0000 osd.2 (osd.2) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:17 smithi192 bash[50362]: cluster 2023-07-10T17:12:16.560352+0000 osd.3 (osd.3) 11 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:17 smithi192 bash[50362]: cluster 2023-07-10T17:12:16.561232+0000 osd.4 (osd.4) 8 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:17 smithi192 bash[50362]: cluster 2023-07-10T17:12:16.561290+0000 osd.5 (osd.5) 9 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:17 smithi192 bash[50362]: cluster 2023-07-10T17:12:16.561397+0000 osd.7 (osd.7) 13 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:17.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:17 smithi192 bash[50362]: cluster 2023-07-10T17:12:16.562998+0000 osd.6 (osd.6) 11 : cluster 3 failed to encode map e84 with expected crc 2023-07-10T17:12:18.191 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:17 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.1.service: Succeeded. 2023-07-10T17:12:18.191 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:17 smithi116 systemd[1]: Stopped Ceph osd.1 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:12:18.191 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:18 smithi116 systemd[1]: Started Ceph osd.1 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:12:18.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:18 smithi192 bash[50362]: cluster 2023-07-10T17:12:17.484985+0000 mgr.y (mgr.25390) 162 : cluster 0 pgmap v93: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2023-07-10T17:12:18.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:18 smithi192 bash[50362]: cluster 2023-07-10T17:12:17.557098+0000 mon.a (mon.0) 350 : cluster 3 Health check failed: Degraded data redundancy: 130/999 objects degraded (13.013%), 22 pgs degraded (PG_DEGRADED) 2023-07-10T17:12:18.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:18 smithi192 bash[50362]: audit 2023-07-10T17:12:18.223232+0000 mon.a (mon.0) 351 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:18.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:18 smithi192 bash[50362]: audit 2023-07-10T17:12:18.229728+0000 mon.a (mon.0) 352 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:18.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:18 smithi116 bash[63900]: cluster 2023-07-10T17:12:17.484985+0000 mgr.y (mgr.25390) 162 : cluster 0 pgmap v93: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2023-07-10T17:12:18.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:18 smithi116 bash[63900]: cluster 2023-07-10T17:12:17.557098+0000 mon.a (mon.0) 350 : cluster 3 Health check failed: Degraded data redundancy: 130/999 objects degraded (13.013%), 22 pgs degraded (PG_DEGRADED) 2023-07-10T17:12:18.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:18 smithi116 bash[63900]: audit 2023-07-10T17:12:18.223232+0000 mon.a (mon.0) 351 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:18.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:18 smithi116 bash[63900]: audit 2023-07-10T17:12:18.229728+0000 mon.a (mon.0) 352 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:18.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:18 smithi116 bash[65980]: cluster 2023-07-10T17:12:17.484985+0000 mgr.y (mgr.25390) 162 : cluster 0 pgmap v93: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2023-07-10T17:12:18.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:18 smithi116 bash[65980]: cluster 2023-07-10T17:12:17.557098+0000 mon.a (mon.0) 350 : cluster 3 Health check failed: Degraded data redundancy: 130/999 objects degraded (13.013%), 22 pgs degraded (PG_DEGRADED) 2023-07-10T17:12:18.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:18 smithi116 bash[65980]: audit 2023-07-10T17:12:18.223232+0000 mon.a (mon.0) 351 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:18.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:18 smithi116 bash[65980]: audit 2023-07-10T17:12:18.229728+0000 mon.a (mon.0) 352 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:20.393 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:20 smithi116 bash[72553]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-07-10T17:12:20.394 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:20 smithi116 bash[72553]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2023-07-10T17:12:20.394 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:20 smithi116 bash[72553]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2023-07-10T17:12:20.394 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:20 smithi116 bash[72553]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2023-07-10T17:12:20.394 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:20 smithi116 bash[72553]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2023-07-10T17:12:20.395 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:20 smithi116 bash[72553]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-07-10T17:12:20.395 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:20 smithi116 bash[72553]: --> ceph-volume raw activate successful for osd ID: 1 2023-07-10T17:12:20.881 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:20 smithi116 bash[65980]: cluster 2023-07-10T17:12:19.485865+0000 mgr.y (mgr.25390) 163 : cluster 0 pgmap v94: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2023-07-10T17:12:20.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:20 smithi116 bash[63900]: cluster 2023-07-10T17:12:19.485865+0000 mgr.y (mgr.25390) 163 : cluster 0 pgmap v94: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2023-07-10T17:12:20.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:20 smithi192 bash[50362]: cluster 2023-07-10T17:12:19.485865+0000 mgr.y (mgr.25390) 163 : cluster 0 pgmap v94: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2023-07-10T17:12:22.143 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:21 smithi116 bash[72834]: debug 2023-07-10T17:12:21.947+0000 7f16af6f8700 1 -- 172.21.15.116:0/3422289989 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5653f2f284e0 con 0x5653f2f82800 2023-07-10T17:12:22.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:22 smithi192 bash[50362]: cluster 2023-07-10T17:12:21.486908+0000 mgr.y (mgr.25390) 164 : cluster 0 pgmap v95: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2023-07-10T17:12:22.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:22 smithi116 bash[63900]: cluster 2023-07-10T17:12:21.486908+0000 mgr.y (mgr.25390) 164 : cluster 0 pgmap v95: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2023-07-10T17:12:22.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:22 smithi116 bash[65980]: cluster 2023-07-10T17:12:21.486908+0000 mgr.y (mgr.25390) 164 : cluster 0 pgmap v95: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2023-07-10T17:12:22.894 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:22 smithi116 bash[72834]: debug 2023-07-10T17:12:22.491+0000 7f16b887d640 -1 Falling back to public interface 2023-07-10T17:12:24.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:24 smithi192 bash[50362]: cluster 2023-07-10T17:12:23.488939+0000 mgr.y (mgr.25390) 165 : cluster 0 pgmap v96: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2023-07-10T17:12:24.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:24 smithi116 bash[63900]: cluster 2023-07-10T17:12:23.488939+0000 mgr.y (mgr.25390) 165 : cluster 0 pgmap v96: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2023-07-10T17:12:24.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:24 smithi116 bash[65980]: cluster 2023-07-10T17:12:23.488939+0000 mgr.y (mgr.25390) 165 : cluster 0 pgmap v96: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2023-07-10T17:12:25.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:25 smithi192 bash[50362]: audit 2023-07-10T17:12:25.561337+0000 mon.c (mon.1) 16 : audit 1 from='osd.1 [v2:172.21.15.116:6810/3733054895,v1:172.21.15.116:6811/3733054895]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-10T17:12:25.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:25 smithi192 bash[50362]: audit 2023-07-10T17:12:25.561758+0000 mon.a (mon.0) 353 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-10T17:12:25.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:25 smithi116 bash[63900]: audit 2023-07-10T17:12:25.561337+0000 mon.c (mon.1) 16 : audit 1 from='osd.1 [v2:172.21.15.116:6810/3733054895,v1:172.21.15.116:6811/3733054895]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-10T17:12:25.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:25 smithi116 bash[63900]: audit 2023-07-10T17:12:25.561758+0000 mon.a (mon.0) 353 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-10T17:12:25.894 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:25 smithi116 bash[72834]: debug 2023-07-10T17:12:25.555+0000 7f16b887d640 -1 osd.1 82 log_to_monitors true 2023-07-10T17:12:25.894 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:12:25 smithi116 bash[72834]: debug 2023-07-10T17:12:25.639+0000 7f16a9cea700 -1 osd.1 82 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-10T17:12:25.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:25 smithi116 bash[65980]: audit 2023-07-10T17:12:25.561337+0000 mon.c (mon.1) 16 : audit 1 from='osd.1 [v2:172.21.15.116:6810/3733054895,v1:172.21.15.116:6811/3733054895]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-10T17:12:25.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:25 smithi116 bash[65980]: audit 2023-07-10T17:12:25.561758+0000 mon.a (mon.0) 353 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-10T17:12:26.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:26 smithi192 bash[50362]: cluster 2023-07-10T17:12:25.489556+0000 mgr.y (mgr.25390) 166 : cluster 0 pgmap v97: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2023-07-10T17:12:26.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:26 smithi192 bash[50362]: cluster 2023-07-10T17:12:25.585763+0000 osd.4 (osd.4) 9 : cluster 3 failed to encode map e85 with expected crc 2023-07-10T17:12:26.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:26 smithi192 bash[50362]: cluster 2023-07-10T17:12:25.586083+0000 osd.6 (osd.6) 12 : cluster 3 failed to encode map e85 with expected crc 2023-07-10T17:12:26.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:26 smithi192 bash[50362]: audit 2023-07-10T17:12:25.587868+0000 mon.a (mon.0) 354 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-10T17:12:26.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:26 smithi192 bash[50362]: cluster 2023-07-10T17:12:25.588070+0000 mon.a (mon.0) 355 : cluster 0 osdmap e85: 8 total, 7 up, 8 in 2023-07-10T17:12:26.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:26 smithi192 bash[50362]: audit 2023-07-10T17:12:25.589167+0000 mon.c (mon.1) 17 : audit 1 from='osd.1 [v2:172.21.15.116:6810/3733054895,v1:172.21.15.116:6811/3733054895]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:12:26.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:26 smithi192 bash[50362]: audit 2023-07-10T17:12:25.589727+0000 mon.a (mon.0) 356 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:12:26.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:26 smithi192 bash[50362]: audit 2023-07-10T17:12:25.993369+0000 mon.a (mon.0) 357 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:26.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:26 smithi192 bash[50362]: audit 2023-07-10T17:12:26.003797+0000 mon.a (mon.0) 358 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:26.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[65980]: cluster 2023-07-10T17:12:25.489556+0000 mgr.y (mgr.25390) 166 : cluster 0 pgmap v97: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2023-07-10T17:12:26.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[65980]: cluster 2023-07-10T17:12:25.585763+0000 osd.4 (osd.4) 9 : cluster 3 failed to encode map e85 with expected crc 2023-07-10T17:12:26.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[65980]: cluster 2023-07-10T17:12:25.586083+0000 osd.6 (osd.6) 12 : cluster 3 failed to encode map e85 with expected crc 2023-07-10T17:12:26.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[65980]: audit 2023-07-10T17:12:25.587868+0000 mon.a (mon.0) 354 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-10T17:12:26.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[65980]: cluster 2023-07-10T17:12:25.588070+0000 mon.a (mon.0) 355 : cluster 0 osdmap e85: 8 total, 7 up, 8 in 2023-07-10T17:12:26.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[65980]: audit 2023-07-10T17:12:25.589167+0000 mon.c (mon.1) 17 : audit 1 from='osd.1 [v2:172.21.15.116:6810/3733054895,v1:172.21.15.116:6811/3733054895]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:12:26.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[65980]: audit 2023-07-10T17:12:25.589727+0000 mon.a (mon.0) 356 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:12:26.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[65980]: audit 2023-07-10T17:12:25.993369+0000 mon.a (mon.0) 357 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:26.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[65980]: audit 2023-07-10T17:12:26.003797+0000 mon.a (mon.0) 358 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:26.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[63900]: cluster 2023-07-10T17:12:25.489556+0000 mgr.y (mgr.25390) 166 : cluster 0 pgmap v97: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2023-07-10T17:12:26.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[63900]: cluster 2023-07-10T17:12:25.585763+0000 osd.4 (osd.4) 9 : cluster 3 failed to encode map e85 with expected crc 2023-07-10T17:12:26.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[63900]: cluster 2023-07-10T17:12:25.586083+0000 osd.6 (osd.6) 12 : cluster 3 failed to encode map e85 with expected crc 2023-07-10T17:12:26.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[63900]: audit 2023-07-10T17:12:25.587868+0000 mon.a (mon.0) 354 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-10T17:12:26.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[63900]: cluster 2023-07-10T17:12:25.588070+0000 mon.a (mon.0) 355 : cluster 0 osdmap e85: 8 total, 7 up, 8 in 2023-07-10T17:12:26.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[63900]: audit 2023-07-10T17:12:25.589167+0000 mon.c (mon.1) 17 : audit 1 from='osd.1 [v2:172.21.15.116:6810/3733054895,v1:172.21.15.116:6811/3733054895]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:12:26.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[63900]: audit 2023-07-10T17:12:25.589727+0000 mon.a (mon.0) 356 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:12:26.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[63900]: audit 2023-07-10T17:12:25.993369+0000 mon.a (mon.0) 357 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:26.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:26 smithi116 bash[63900]: audit 2023-07-10T17:12:26.003797+0000 mon.a (mon.0) 358 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:27.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:27 smithi192 bash[50362]: cluster 2023-07-10T17:12:26.584446+0000 mon.a (mon.0) 359 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:12:27.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:27 smithi192 bash[50362]: cluster 2023-07-10T17:12:26.593405+0000 osd.2 (osd.2) 9 : cluster 3 failed to encode map e86 with expected crc 2023-07-10T17:12:27.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:27 smithi192 bash[50362]: cluster 2023-07-10T17:12:26.597187+0000 osd.6 (osd.6) 13 : cluster 3 failed to encode map e86 with expected crc 2023-07-10T17:12:27.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:27 smithi192 bash[50362]: cluster 2023-07-10T17:12:26.597588+0000 mon.a (mon.0) 360 : cluster 1 osd.1 [v2:172.21.15.116:6810/3733054895,v1:172.21.15.116:6811/3733054895] boot 2023-07-10T17:12:27.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:27 smithi192 bash[50362]: cluster 2023-07-10T17:12:26.597673+0000 mon.a (mon.0) 361 : cluster 0 osdmap e86: 8 total, 8 up, 8 in 2023-07-10T17:12:27.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:27 smithi192 bash[50362]: cluster 2023-07-10T17:12:26.598890+0000 osd.3 (osd.3) 12 : cluster 3 failed to encode map e86 with expected crc 2023-07-10T17:12:27.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:27 smithi192 bash[50362]: audit 2023-07-10T17:12:26.599368+0000 mon.a (mon.0) 362 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T17:12:27.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:27 smithi192 bash[50362]: cluster 2023-07-10T17:12:26.600270+0000 osd.5 (osd.5) 10 : cluster 3 failed to encode map e86 with expected crc 2023-07-10T17:12:27.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:27 smithi192 bash[50362]: cluster 2023-07-10T17:12:26.600531+0000 osd.4 (osd.4) 10 : cluster 3 failed to encode map e86 with expected crc 2023-07-10T17:12:27.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:27 smithi192 bash[50362]: audit 2023-07-10T17:12:26.801654+0000 mon.a (mon.0) 363 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:27.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:27 smithi192 bash[50362]: audit 2023-07-10T17:12:26.811035+0000 mon.a (mon.0) 364 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:27.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[63900]: cluster 2023-07-10T17:12:26.584446+0000 mon.a (mon.0) 359 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:12:27.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[63900]: cluster 2023-07-10T17:12:26.593405+0000 osd.2 (osd.2) 9 : cluster 3 failed to encode map e86 with expected crc 2023-07-10T17:12:27.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[63900]: cluster 2023-07-10T17:12:26.597187+0000 osd.6 (osd.6) 13 : cluster 3 failed to encode map e86 with expected crc 2023-07-10T17:12:27.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[63900]: cluster 2023-07-10T17:12:26.597588+0000 mon.a (mon.0) 360 : cluster 1 osd.1 [v2:172.21.15.116:6810/3733054895,v1:172.21.15.116:6811/3733054895] boot 2023-07-10T17:12:27.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[63900]: cluster 2023-07-10T17:12:26.597673+0000 mon.a (mon.0) 361 : cluster 0 osdmap e86: 8 total, 8 up, 8 in 2023-07-10T17:12:27.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[63900]: cluster 2023-07-10T17:12:26.598890+0000 osd.3 (osd.3) 12 : cluster 3 failed to encode map e86 with expected crc 2023-07-10T17:12:27.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[63900]: audit 2023-07-10T17:12:26.599368+0000 mon.a (mon.0) 362 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T17:12:27.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[63900]: cluster 2023-07-10T17:12:26.600270+0000 osd.5 (osd.5) 10 : cluster 3 failed to encode map e86 with expected crc 2023-07-10T17:12:27.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[63900]: cluster 2023-07-10T17:12:26.600531+0000 osd.4 (osd.4) 10 : cluster 3 failed to encode map e86 with expected crc 2023-07-10T17:12:27.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[63900]: audit 2023-07-10T17:12:26.801654+0000 mon.a (mon.0) 363 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:27.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[63900]: audit 2023-07-10T17:12:26.811035+0000 mon.a (mon.0) 364 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:27.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[65980]: cluster 2023-07-10T17:12:26.584446+0000 mon.a (mon.0) 359 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:12:27.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[65980]: cluster 2023-07-10T17:12:26.593405+0000 osd.2 (osd.2) 9 : cluster 3 failed to encode map e86 with expected crc 2023-07-10T17:12:27.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[65980]: cluster 2023-07-10T17:12:26.597187+0000 osd.6 (osd.6) 13 : cluster 3 failed to encode map e86 with expected crc 2023-07-10T17:12:27.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[65980]: cluster 2023-07-10T17:12:26.597588+0000 mon.a (mon.0) 360 : cluster 1 osd.1 [v2:172.21.15.116:6810/3733054895,v1:172.21.15.116:6811/3733054895] boot 2023-07-10T17:12:27.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[65980]: cluster 2023-07-10T17:12:26.597673+0000 mon.a (mon.0) 361 : cluster 0 osdmap e86: 8 total, 8 up, 8 in 2023-07-10T17:12:27.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[65980]: cluster 2023-07-10T17:12:26.598890+0000 osd.3 (osd.3) 12 : cluster 3 failed to encode map e86 with expected crc 2023-07-10T17:12:27.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[65980]: audit 2023-07-10T17:12:26.599368+0000 mon.a (mon.0) 362 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-10T17:12:27.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[65980]: cluster 2023-07-10T17:12:26.600270+0000 osd.5 (osd.5) 10 : cluster 3 failed to encode map e86 with expected crc 2023-07-10T17:12:27.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[65980]: cluster 2023-07-10T17:12:26.600531+0000 osd.4 (osd.4) 10 : cluster 3 failed to encode map e86 with expected crc 2023-07-10T17:12:27.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[65980]: audit 2023-07-10T17:12:26.801654+0000 mon.a (mon.0) 363 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:27.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:27 smithi116 bash[65980]: audit 2023-07-10T17:12:26.811035+0000 mon.a (mon.0) 364 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:28.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:28 smithi192 bash[50362]: cluster 2023-07-10T17:12:27.491081+0000 mgr.y (mgr.25390) 167 : cluster 0 pgmap v100: 161 pgs: 36 peering, 8 active+undersized+degraded, 19 active+undersized, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 54/999 objects degraded (5.405%) 2023-07-10T17:12:28.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:28 smithi192 bash[50362]: cluster 2023-07-10T17:12:27.615697+0000 osd.5 (osd.5) 11 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:28 smithi192 bash[50362]: cluster 2023-07-10T17:12:27.617059+0000 osd.6 (osd.6) 14 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:28 smithi192 bash[50362]: cluster 2023-07-10T17:12:27.617813+0000 mon.a (mon.0) 365 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2023-07-10T17:12:28.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:28 smithi192 bash[50362]: cluster 2023-07-10T17:12:27.618134+0000 osd.3 (osd.3) 13 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:28 smithi192 bash[50362]: cluster 2023-07-10T17:12:27.618183+0000 osd.2 (osd.2) 10 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:28 smithi192 bash[50362]: cluster 2023-07-10T17:12:27.618893+0000 osd.5 (osd.5) 12 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:28 smithi192 bash[50362]: cluster 2023-07-10T17:12:27.620471+0000 osd.6 (osd.6) 15 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:28 smithi192 bash[50362]: cluster 2023-07-10T17:12:27.620702+0000 osd.4 (osd.4) 11 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:28 smithi192 bash[50362]: cluster 2023-07-10T17:12:27.620925+0000 osd.7 (osd.7) 14 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:28 smithi192 bash[50362]: cluster 2023-07-10T17:12:27.629074+0000 osd.7 (osd.7) 15 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:28 smithi192 bash[50362]: cluster 2023-07-10T17:12:27.631963+0000 osd.7 (osd.7) 16 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:28 smithi192 bash[50362]: cluster 2023-07-10T17:12:27.806538+0000 mon.a (mon.0) 366 : cluster 3 Health check update: Degraded data redundancy: 54/999 objects degraded (5.405%), 8 pgs degraded (PG_DEGRADED) 2023-07-10T17:12:28.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[63900]: cluster 2023-07-10T17:12:27.491081+0000 mgr.y (mgr.25390) 167 : cluster 0 pgmap v100: 161 pgs: 36 peering, 8 active+undersized+degraded, 19 active+undersized, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 54/999 objects degraded (5.405%) 2023-07-10T17:12:28.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[63900]: cluster 2023-07-10T17:12:27.615697+0000 osd.5 (osd.5) 11 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[63900]: cluster 2023-07-10T17:12:27.617059+0000 osd.6 (osd.6) 14 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[63900]: cluster 2023-07-10T17:12:27.617813+0000 mon.a (mon.0) 365 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2023-07-10T17:12:28.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[63900]: cluster 2023-07-10T17:12:27.618134+0000 osd.3 (osd.3) 13 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[63900]: cluster 2023-07-10T17:12:27.618183+0000 osd.2 (osd.2) 10 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[63900]: cluster 2023-07-10T17:12:27.618893+0000 osd.5 (osd.5) 12 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[63900]: cluster 2023-07-10T17:12:27.620471+0000 osd.6 (osd.6) 15 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[63900]: cluster 2023-07-10T17:12:27.620702+0000 osd.4 (osd.4) 11 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[63900]: cluster 2023-07-10T17:12:27.620925+0000 osd.7 (osd.7) 14 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[63900]: cluster 2023-07-10T17:12:27.629074+0000 osd.7 (osd.7) 15 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[63900]: cluster 2023-07-10T17:12:27.631963+0000 osd.7 (osd.7) 16 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[63900]: cluster 2023-07-10T17:12:27.806538+0000 mon.a (mon.0) 366 : cluster 3 Health check update: Degraded data redundancy: 54/999 objects degraded (5.405%), 8 pgs degraded (PG_DEGRADED) 2023-07-10T17:12:28.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[65980]: cluster 2023-07-10T17:12:27.491081+0000 mgr.y (mgr.25390) 167 : cluster 0 pgmap v100: 161 pgs: 36 peering, 8 active+undersized+degraded, 19 active+undersized, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 54/999 objects degraded (5.405%) 2023-07-10T17:12:28.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[65980]: cluster 2023-07-10T17:12:27.615697+0000 osd.5 (osd.5) 11 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[65980]: cluster 2023-07-10T17:12:27.617059+0000 osd.6 (osd.6) 14 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[65980]: cluster 2023-07-10T17:12:27.617813+0000 mon.a (mon.0) 365 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2023-07-10T17:12:28.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[65980]: cluster 2023-07-10T17:12:27.618134+0000 osd.3 (osd.3) 13 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[65980]: cluster 2023-07-10T17:12:27.618183+0000 osd.2 (osd.2) 10 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[65980]: cluster 2023-07-10T17:12:27.618893+0000 osd.5 (osd.5) 12 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[65980]: cluster 2023-07-10T17:12:27.620471+0000 osd.6 (osd.6) 15 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[65980]: cluster 2023-07-10T17:12:27.620702+0000 osd.4 (osd.4) 11 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[65980]: cluster 2023-07-10T17:12:27.620925+0000 osd.7 (osd.7) 14 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[65980]: cluster 2023-07-10T17:12:27.629074+0000 osd.7 (osd.7) 15 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[65980]: cluster 2023-07-10T17:12:27.631963+0000 osd.7 (osd.7) 16 : cluster 3 failed to encode map e87 with expected crc 2023-07-10T17:12:28.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:28 smithi116 bash[65980]: cluster 2023-07-10T17:12:27.806538+0000 mon.a (mon.0) 366 : cluster 3 Health check update: Degraded data redundancy: 54/999 objects degraded (5.405%), 8 pgs degraded (PG_DEGRADED) 2023-07-10T17:12:30.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:30 smithi192 bash[50362]: cluster 2023-07-10T17:12:29.492012+0000 mgr.y (mgr.25390) 168 : cluster 0 pgmap v102: 161 pgs: 36 peering, 8 active+undersized+degraded, 19 active+undersized, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 54/999 objects degraded (5.405%) 2023-07-10T17:12:30.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:30 smithi116 bash[63900]: cluster 2023-07-10T17:12:29.492012+0000 mgr.y (mgr.25390) 168 : cluster 0 pgmap v102: 161 pgs: 36 peering, 8 active+undersized+degraded, 19 active+undersized, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 54/999 objects degraded (5.405%) 2023-07-10T17:12:30.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:30 smithi116 bash[65980]: cluster 2023-07-10T17:12:29.492012+0000 mgr.y (mgr.25390) 168 : cluster 0 pgmap v102: 161 pgs: 36 peering, 8 active+undersized+degraded, 19 active+undersized, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 54/999 objects degraded (5.405%) 2023-07-10T17:12:32.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:32 smithi192 bash[50362]: cluster 2023-07-10T17:12:31.493064+0000 mgr.y (mgr.25390) 169 : cluster 0 pgmap v103: 161 pgs: 36 peering, 1 active+undersized+degraded, 5 active+undersized, 119 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 10/999 objects degraded (1.001%) 2023-07-10T17:12:32.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:32 smithi116 bash[63900]: cluster 2023-07-10T17:12:31.493064+0000 mgr.y (mgr.25390) 169 : cluster 0 pgmap v103: 161 pgs: 36 peering, 1 active+undersized+degraded, 5 active+undersized, 119 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 10/999 objects degraded (1.001%) 2023-07-10T17:12:32.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:32 smithi116 bash[65980]: cluster 2023-07-10T17:12:31.493064+0000 mgr.y (mgr.25390) 169 : cluster 0 pgmap v103: 161 pgs: 36 peering, 1 active+undersized+degraded, 5 active+undersized, 119 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 10/999 objects degraded (1.001%) 2023-07-10T17:12:33.987 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:33 smithi116 bash[63900]: cluster 2023-07-10T17:12:33.625248+0000 mon.a (mon.0) 367 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/999 objects degraded (1.001%), 1 pg degraded) 2023-07-10T17:12:33.988 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:33 smithi116 bash[65980]: cluster 2023-07-10T17:12:33.625248+0000 mon.a (mon.0) 367 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/999 objects degraded (1.001%), 1 pg degraded) 2023-07-10T17:12:34.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:33 smithi192 bash[50362]: cluster 2023-07-10T17:12:33.625248+0000 mon.a (mon.0) 367 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/999 objects degraded (1.001%), 1 pg degraded) 2023-07-10T17:12:35.084 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:34 smithi116 bash[65980]: cluster 2023-07-10T17:12:33.494742+0000 mgr.y (mgr.25390) 170 : cluster 0 pgmap v104: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:35.084 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:34 smithi116 bash[63900]: cluster 2023-07-10T17:12:33.494742+0000 mgr.y (mgr.25390) 170 : cluster 0 pgmap v104: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:35.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:34 smithi192 bash[50362]: cluster 2023-07-10T17:12:33.494742+0000 mgr.y (mgr.25390) 170 : cluster 0 pgmap v104: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:36.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:36 smithi116 bash[63900]: cluster 2023-07-10T17:12:35.495536+0000 mgr.y (mgr.25390) 171 : cluster 0 pgmap v105: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:36.893 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:36 smithi116 bash[65980]: cluster 2023-07-10T17:12:35.495536+0000 mgr.y (mgr.25390) 171 : cluster 0 pgmap v105: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:37.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:36 smithi192 bash[50362]: cluster 2023-07-10T17:12:35.495536+0000 mgr.y (mgr.25390) 171 : cluster 0 pgmap v105: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:12:38.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:38 smithi192 bash[50362]: audit 2023-07-10T17:12:37.363414+0000 mon.a (mon.0) 368 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:38.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:38 smithi192 bash[50362]: audit 2023-07-10T17:12:37.374029+0000 mon.a (mon.0) 369 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:38.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:38 smithi192 bash[50362]: audit 2023-07-10T17:12:37.375736+0000 mon.a (mon.0) 370 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:12:38.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:38 smithi192 bash[50362]: audit 2023-07-10T17:12:37.377012+0000 mon.a (mon.0) 371 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:12:38.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:38 smithi192 bash[50362]: cephadm 2023-07-10T17:12:37.378124+0000 mgr.y (mgr.25390) 172 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:12:38.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:38 smithi192 bash[50362]: cluster 2023-07-10T17:12:37.379756+0000 mgr.y (mgr.25390) 173 : cluster 0 pgmap v106: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-10T17:12:38.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:38 smithi192 bash[50362]: cluster 2023-07-10T17:12:37.380100+0000 mgr.y (mgr.25390) 174 : cluster 0 pgmap v107: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-10T17:12:38.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:38 smithi192 bash[50362]: audit 2023-07-10T17:12:37.385722+0000 mon.a (mon.0) 372 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:38.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:38 smithi192 bash[50362]: audit 2023-07-10T17:12:37.472275+0000 mon.a (mon.0) 373 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:12:38.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:38 smithi192 bash[50362]: audit 2023-07-10T17:12:37.478148+0000 mon.a (mon.0) 374 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:38.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:38 smithi192 bash[50362]: audit 2023-07-10T17:12:37.481063+0000 mon.a (mon.0) 375 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:38.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:38 smithi192 bash[50362]: audit 2023-07-10T17:12:37.483153+0000 mon.a (mon.0) 376 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:38.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:38 smithi192 bash[50362]: audit 2023-07-10T17:12:37.486192+0000 mon.a (mon.0) 377 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-07-10T17:12:38.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:38 smithi192 bash[50362]: audit 2023-07-10T17:12:37.486503+0000 mgr.y (mgr.25390) 175 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-07-10T17:12:38.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:38 smithi192 bash[50362]: cephadm 2023-07-10T17:12:37.488448+0000 mgr.y (mgr.25390) 176 : cephadm 1 Upgrade: osd.2 is safe to restart 2023-07-10T17:12:38.643 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[65980]: audit 2023-07-10T17:12:37.363414+0000 mon.a (mon.0) 368 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:38.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[65980]: audit 2023-07-10T17:12:37.374029+0000 mon.a (mon.0) 369 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:38.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[65980]: audit 2023-07-10T17:12:37.375736+0000 mon.a (mon.0) 370 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:12:38.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[65980]: audit 2023-07-10T17:12:37.377012+0000 mon.a (mon.0) 371 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:12:38.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[65980]: cephadm 2023-07-10T17:12:37.378124+0000 mgr.y (mgr.25390) 172 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:12:38.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[65980]: cluster 2023-07-10T17:12:37.379756+0000 mgr.y (mgr.25390) 173 : cluster 0 pgmap v106: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-10T17:12:38.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[65980]: cluster 2023-07-10T17:12:37.380100+0000 mgr.y (mgr.25390) 174 : cluster 0 pgmap v107: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-10T17:12:38.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[65980]: audit 2023-07-10T17:12:37.385722+0000 mon.a (mon.0) 372 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:38.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[65980]: audit 2023-07-10T17:12:37.472275+0000 mon.a (mon.0) 373 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:12:38.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[65980]: audit 2023-07-10T17:12:37.478148+0000 mon.a (mon.0) 374 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:38.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[65980]: audit 2023-07-10T17:12:37.481063+0000 mon.a (mon.0) 375 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:38.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[65980]: audit 2023-07-10T17:12:37.483153+0000 mon.a (mon.0) 376 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:38.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[65980]: audit 2023-07-10T17:12:37.486192+0000 mon.a (mon.0) 377 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-07-10T17:12:38.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[65980]: audit 2023-07-10T17:12:37.486503+0000 mgr.y (mgr.25390) 175 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-07-10T17:12:38.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[65980]: cephadm 2023-07-10T17:12:37.488448+0000 mgr.y (mgr.25390) 176 : cephadm 1 Upgrade: osd.2 is safe to restart 2023-07-10T17:12:38.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[63900]: audit 2023-07-10T17:12:37.363414+0000 mon.a (mon.0) 368 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:38.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[63900]: audit 2023-07-10T17:12:37.374029+0000 mon.a (mon.0) 369 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:38.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[63900]: audit 2023-07-10T17:12:37.375736+0000 mon.a (mon.0) 370 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:12:38.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[63900]: audit 2023-07-10T17:12:37.377012+0000 mon.a (mon.0) 371 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:12:38.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[63900]: cephadm 2023-07-10T17:12:37.378124+0000 mgr.y (mgr.25390) 172 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:12:38.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[63900]: cluster 2023-07-10T17:12:37.379756+0000 mgr.y (mgr.25390) 173 : cluster 0 pgmap v106: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-10T17:12:38.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[63900]: cluster 2023-07-10T17:12:37.380100+0000 mgr.y (mgr.25390) 174 : cluster 0 pgmap v107: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-10T17:12:38.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[63900]: audit 2023-07-10T17:12:37.385722+0000 mon.a (mon.0) 372 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:38.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[63900]: audit 2023-07-10T17:12:37.472275+0000 mon.a (mon.0) 373 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:12:38.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[63900]: audit 2023-07-10T17:12:37.478148+0000 mon.a (mon.0) 374 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:38.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[63900]: audit 2023-07-10T17:12:37.481063+0000 mon.a (mon.0) 375 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:38.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[63900]: audit 2023-07-10T17:12:37.483153+0000 mon.a (mon.0) 376 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:38.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[63900]: audit 2023-07-10T17:12:37.486192+0000 mon.a (mon.0) 377 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-07-10T17:12:38.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[63900]: audit 2023-07-10T17:12:37.486503+0000 mgr.y (mgr.25390) 175 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-07-10T17:12:38.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:38 smithi116 bash[63900]: cephadm 2023-07-10T17:12:37.488448+0000 mgr.y (mgr.25390) 176 : cephadm 1 Upgrade: osd.2 is safe to restart 2023-07-10T17:12:38.969 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:12:39.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:39 smithi192 bash[50362]: audit 2023-07-10T17:12:38.950915+0000 mgr.y (mgr.25390) 177 : audit 0 from='client.44353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:39.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:39 smithi116 bash[63900]: audit 2023-07-10T17:12:38.950915+0000 mgr.y (mgr.25390) 177 : audit 0 from='client.44353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:39.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:39 smithi116 bash[65980]: audit 2023-07-10T17:12:38.950915+0000 mgr.y (mgr.25390) 177 : audit 0 from='client.44353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:39.715 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:12:39.715 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (5m) 13s ago 12m 10.6M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:12:39.715 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (4m) 114s ago 11m 25.6M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:12:39.715 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 error 13s ago 9m - - 2023-07-10T17:12:39.715 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (3m) 114s ago 16m 428M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:12:39.716 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (4m) 13s ago 18m 472M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:12:39.716 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (3m) 13s ago 18m 57.2M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:12:39.716 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (2m) 114s ago 16m 40.2M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 74a082a2e795 2023-07-10T17:12:39.716 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (2m) 13s ago 16m 49.4M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:12:39.716 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (5m) 13s ago 12m 9440k - 0.18.1 e5a616e4b9cf 3bd058d91353 2023-07-10T17:12:39.716 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (5m) 114s ago 12m 3607k - 0.18.1 e5a616e4b9cf 63551615c679 2023-07-10T17:12:39.716 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (48s) 13s ago 16m 48.8M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 0562f587fbca 2023-07-10T17:12:39.717 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (18s) 13s ago 15m - 4096M 18.1.2-30-g8507faa7 17e8f0d79247 5b2a12a0bfc9 2023-07-10T17:12:39.717 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (15m) 13s ago 15m 51.2M 4096M 16.2.0 24ecd6d5f14c f1fd77b3c454 2023-07-10T17:12:39.717 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (14m) 13s ago 14m 57.0M 4096M 16.2.0 24ecd6d5f14c 60949d479c11 2023-07-10T17:12:39.717 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (14m) 114s ago 14m 55.1M 4096M 16.2.0 24ecd6d5f14c 75fc0b36aab8 2023-07-10T17:12:39.717 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (13m) 114s ago 13m 52.8M 4096M 16.2.0 24ecd6d5f14c d673065333dd 2023-07-10T17:12:39.717 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (13m) 114s ago 13m 53.7M 4096M 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:12:39.717 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (13m) 114s ago 13m 56.7M 4096M 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:12:39.718 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 error 114s ago 12m - - 2023-07-10T17:12:39.718 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (10m) 13s ago 10m 55.3M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:12:39.718 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (10m) 114s ago 10m 54.8M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:12:39.718 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (9m) 13s ago 9m 55.2M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:12:39.718 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (9m) 114s ago 9m 56.6M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:12:40.159 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:12:40.159 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:12:40.160 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:12:40.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:12:40.160 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:12:40.160 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:12:40.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:12:40.160 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:12:40.160 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 6, 2023-07-10T17:12:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:12:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:12:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:12:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:12:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:12:40.162 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:12:40.162 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 10, 2023-07-10T17:12:40.162 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 7 2023-07-10T17:12:40.162 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:12:40.162 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:12:40.530 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:12:40.530 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:12:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:12:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:12:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:12:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2023-07-10T17:12:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2023-07-10T17:12:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:12:40.532 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "7/23 daemons upgraded", 2023-07-10T17:12:40.532 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading osd daemons", 2023-07-10T17:12:40.532 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:12:40.532 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:12:40.972 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:12:40.972 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:12:40.972 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:12:40.972 INFO:teuthology.orchestra.run.smithi116.stdout: daemon prometheus.a on smithi192 is in error state 2023-07-10T17:12:41.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[63900]: audit 2023-07-10T17:12:39.324422+0000 mgr.y (mgr.25390) 178 : audit 0 from='client.44359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:41.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[63900]: cluster 2023-07-10T17:12:39.380762+0000 mgr.y (mgr.25390) 179 : cluster 0 pgmap v108: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-10T17:12:41.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[63900]: audit 2023-07-10T17:12:39.685122+0000 mgr.y (mgr.25390) 180 : audit 0 from='client.44365 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:41.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[63900]: cephadm 2023-07-10T17:12:39.909476+0000 mgr.y (mgr.25390) 181 : cephadm 1 Upgrade: Updating osd.2 2023-07-10T17:12:41.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[63900]: audit 2023-07-10T17:12:39.918338+0000 mon.a (mon.0) 378 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:41.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[63900]: audit 2023-07-10T17:12:39.920171+0000 mon.a (mon.0) 379 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-10T17:12:41.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[63900]: audit 2023-07-10T17:12:39.921586+0000 mon.a (mon.0) 380 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:12:41.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[63900]: cephadm 2023-07-10T17:12:39.924780+0000 mgr.y (mgr.25390) 182 : cephadm 1 Deploying daemon osd.2 on smithi116 2023-07-10T17:12:41.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[63900]: audit 2023-07-10T17:12:40.155113+0000 mon.a (mon.0) 381 : audit 0 from='client.44371 172.21.15.116:0/1586901374' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:41.276 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[65980]: audit 2023-07-10T17:12:39.324422+0000 mgr.y (mgr.25390) 178 : audit 0 from='client.44359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:41.276 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[65980]: cluster 2023-07-10T17:12:39.380762+0000 mgr.y (mgr.25390) 179 : cluster 0 pgmap v108: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-10T17:12:41.277 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[65980]: audit 2023-07-10T17:12:39.685122+0000 mgr.y (mgr.25390) 180 : audit 0 from='client.44365 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:41.277 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[65980]: cephadm 2023-07-10T17:12:39.909476+0000 mgr.y (mgr.25390) 181 : cephadm 1 Upgrade: Updating osd.2 2023-07-10T17:12:41.277 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[65980]: audit 2023-07-10T17:12:39.918338+0000 mon.a (mon.0) 378 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:41.278 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[65980]: audit 2023-07-10T17:12:39.920171+0000 mon.a (mon.0) 379 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-10T17:12:41.278 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[65980]: audit 2023-07-10T17:12:39.921586+0000 mon.a (mon.0) 380 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:12:41.278 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[65980]: cephadm 2023-07-10T17:12:39.924780+0000 mgr.y (mgr.25390) 182 : cephadm 1 Deploying daemon osd.2 on smithi116 2023-07-10T17:12:41.278 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:40 smithi116 bash[65980]: audit 2023-07-10T17:12:40.155113+0000 mon.a (mon.0) 381 : audit 0 from='client.44371 172.21.15.116:0/1586901374' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:41.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:40 smithi192 bash[50362]: audit 2023-07-10T17:12:39.324422+0000 mgr.y (mgr.25390) 178 : audit 0 from='client.44359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:41.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:40 smithi192 bash[50362]: cluster 2023-07-10T17:12:39.380762+0000 mgr.y (mgr.25390) 179 : cluster 0 pgmap v108: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-10T17:12:41.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:40 smithi192 bash[50362]: audit 2023-07-10T17:12:39.685122+0000 mgr.y (mgr.25390) 180 : audit 0 from='client.44365 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:41.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:40 smithi192 bash[50362]: cephadm 2023-07-10T17:12:39.909476+0000 mgr.y (mgr.25390) 181 : cephadm 1 Upgrade: Updating osd.2 2023-07-10T17:12:41.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:40 smithi192 bash[50362]: audit 2023-07-10T17:12:39.918338+0000 mon.a (mon.0) 378 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:41.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:40 smithi192 bash[50362]: audit 2023-07-10T17:12:39.920171+0000 mon.a (mon.0) 379 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-10T17:12:41.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:40 smithi192 bash[50362]: audit 2023-07-10T17:12:39.921586+0000 mon.a (mon.0) 380 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:12:41.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:40 smithi192 bash[50362]: cephadm 2023-07-10T17:12:39.924780+0000 mgr.y (mgr.25390) 182 : cephadm 1 Deploying daemon osd.2 on smithi116 2023-07-10T17:12:41.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:40 smithi192 bash[50362]: audit 2023-07-10T17:12:40.155113+0000 mon.a (mon.0) 381 : audit 0 from='client.44371 172.21.15.116:0/1586901374' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:12:42.318 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:41 smithi116 bash[65980]: audit 2023-07-10T17:12:40.526018+0000 mgr.y (mgr.25390) 183 : audit 0 from='client.44377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:42.318 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:41 smithi116 bash[65980]: audit 2023-07-10T17:12:40.968065+0000 mon.c (mon.1) 18 : audit 0 from='client.44383 172.21.15.116:0/4163022919' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:12:42.318 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:41 smithi116 bash[65980]: audit 2023-07-10T17:12:41.409627+0000 mon.a (mon.0) 382 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:42.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:41 smithi116 bash[63900]: audit 2023-07-10T17:12:40.526018+0000 mgr.y (mgr.25390) 183 : audit 0 from='client.44377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:42.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:41 smithi116 bash[63900]: audit 2023-07-10T17:12:40.968065+0000 mon.c (mon.1) 18 : audit 0 from='client.44383 172.21.15.116:0/4163022919' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:12:42.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:41 smithi116 bash[63900]: audit 2023-07-10T17:12:41.409627+0000 mon.a (mon.0) 382 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:42.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:41 smithi192 bash[50362]: audit 2023-07-10T17:12:40.526018+0000 mgr.y (mgr.25390) 183 : audit 0 from='client.44377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:12:42.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:41 smithi192 bash[50362]: audit 2023-07-10T17:12:40.968065+0000 mon.c (mon.1) 18 : audit 0 from='client.44383 172.21.15.116:0/4163022919' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:12:42.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:41 smithi192 bash[50362]: audit 2023-07-10T17:12:41.409627+0000 mon.a (mon.0) 382 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:43.265 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:42 smithi116 bash[65980]: cluster 2023-07-10T17:12:41.381400+0000 mgr.y (mgr.25390) 184 : cluster 0 pgmap v109: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2023-07-10T17:12:43.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:42 smithi116 bash[63900]: cluster 2023-07-10T17:12:41.381400+0000 mgr.y (mgr.25390) 184 : cluster 0 pgmap v109: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2023-07-10T17:12:43.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:42 smithi192 bash[50362]: cluster 2023-07-10T17:12:41.381400+0000 mgr.y (mgr.25390) 184 : cluster 0 pgmap v109: 161 pgs: 161 active+clean; 604 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2023-07-10T17:12:44.756 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:44 smithi116 systemd[1]: Stopping Ceph osd.2 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:12:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:44 smithi116 bash[63900]: cluster 2023-07-10T17:12:43.382939+0000 mgr.y (mgr.25390) 185 : cluster 0 pgmap v110: 161 pgs: 161 active+clean; 604 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2023-07-10T17:12:45.009 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:44 smithi116 bash[65980]: cluster 2023-07-10T17:12:43.382939+0000 mgr.y (mgr.25390) 185 : cluster 0 pgmap v110: 161 pgs: 161 active+clean; 604 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2023-07-10T17:12:45.009 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:44 smithi116 bash[36504]: debug 2023-07-10T17:12:44.750+0000 7f7e536a8700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:12:45.009 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:44 smithi116 bash[36504]: debug 2023-07-10T17:12:44.750+0000 7f7e536a8700 -1 osd.2 87 *** Got signal Terminated *** 2023-07-10T17:12:45.010 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:44 smithi116 bash[36504]: debug 2023-07-10T17:12:44.750+0000 7f7e536a8700 -1 osd.2 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-10T17:12:45.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:44 smithi192 bash[50362]: cluster 2023-07-10T17:12:43.382939+0000 mgr.y (mgr.25390) 185 : cluster 0 pgmap v110: 161 pgs: 161 active+clean; 604 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2023-07-10T17:12:45.893 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[74607]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-osd.2 2023-07-10T17:12:46.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.021699+0000 mon.a (mon.0) 383 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.021883+0000 mon.a (mon.0) 384 : cluster 1 osd.2 failed (root=default,host=smithi116) (connection refused reported by osd.5) 2023-07-10T17:12:46.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.022611+0000 mon.a (mon.0) 385 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.023419+0000 mon.a (mon.0) 386 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.024036+0000 mon.a (mon.0) 387 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-10T17:12:46.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.024436+0000 mon.a (mon.0) 388 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.024599+0000 mon.a (mon.0) 389 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.024850+0000 mon.a (mon.0) 390 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.025021+0000 mon.a (mon.0) 391 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-10T17:12:46.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.025315+0000 mon.a (mon.0) 392 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.025416+0000 mon.a (mon.0) 393 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.025687+0000 mon.a (mon.0) 394 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-10T17:12:46.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.025901+0000 mon.a (mon.0) 395 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-10T17:12:46.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.026264+0000 mon.a (mon.0) 396 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-10T17:12:46.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.026580+0000 mon.a (mon.0) 397 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-10T17:12:46.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.359977+0000 mon.a (mon.0) 398 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.360140+0000 mon.a (mon.0) 399 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.360527+0000 mon.a (mon.0) 400 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.361050+0000 mon.a (mon.0) 401 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.427246+0000 mon.a (mon.0) 402 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.427730+0000 mon.a (mon.0) 403 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.428372+0000 mon.a (mon.0) 404 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-10T17:12:46.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.428794+0000 mon.a (mon.0) 405 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.429093+0000 mon.a (mon.0) 406 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.429308+0000 mon.a (mon.0) 407 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.151 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.429601+0000 mon.a (mon.0) 408 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-10T17:12:46.151 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.021699+0000 mon.a (mon.0) 383 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.151 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.021883+0000 mon.a (mon.0) 384 : cluster 1 osd.2 failed (root=default,host=smithi116) (connection refused reported by osd.5) 2023-07-10T17:12:46.151 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.022611+0000 mon.a (mon.0) 385 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.151 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.023419+0000 mon.a (mon.0) 386 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.152 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.024036+0000 mon.a (mon.0) 387 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-10T17:12:46.152 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.024436+0000 mon.a (mon.0) 388 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.152 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.024599+0000 mon.a (mon.0) 389 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.152 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.024850+0000 mon.a (mon.0) 390 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.152 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.025021+0000 mon.a (mon.0) 391 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-10T17:12:46.153 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.025315+0000 mon.a (mon.0) 392 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.153 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.025416+0000 mon.a (mon.0) 393 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.153 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.025687+0000 mon.a (mon.0) 394 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-10T17:12:46.153 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.025901+0000 mon.a (mon.0) 395 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-10T17:12:46.153 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.026264+0000 mon.a (mon.0) 396 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-10T17:12:46.153 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.026580+0000 mon.a (mon.0) 397 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-10T17:12:46.153 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.359977+0000 mon.a (mon.0) 398 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.154 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.360140+0000 mon.a (mon.0) 399 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.154 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.360527+0000 mon.a (mon.0) 400 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.154 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.361050+0000 mon.a (mon.0) 401 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.154 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.427246+0000 mon.a (mon.0) 402 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.154 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.427730+0000 mon.a (mon.0) 403 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.154 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.428372+0000 mon.a (mon.0) 404 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-10T17:12:46.154 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.428794+0000 mon.a (mon.0) 405 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.154 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.429093+0000 mon.a (mon.0) 406 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.429308+0000 mon.a (mon.0) 407 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.429601+0000 mon.a (mon.0) 408 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-10T17:12:46.155 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.429843+0000 mon.a (mon.0) 409 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.155 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.430127+0000 mon.a (mon.0) 410 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.155 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.430234+0000 mon.a (mon.0) 411 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-10T17:12:46.156 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.430447+0000 mon.a (mon.0) 412 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.156 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.430600+0000 mon.a (mon.0) 413 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-10T17:12:46.156 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.430850+0000 mon.a (mon.0) 414 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-10T17:12:46.156 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.431067+0000 mon.a (mon.0) 415 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-10T17:12:46.156 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.563741+0000 mon.a (mon.0) 416 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.156 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.565409+0000 mon.a (mon.0) 417 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.156 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.565545+0000 mon.a (mon.0) 418 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.157 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.565829+0000 mon.a (mon.0) 419 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.429843+0000 mon.a (mon.0) 409 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.430127+0000 mon.a (mon.0) 410 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.430234+0000 mon.a (mon.0) 411 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-10T17:12:46.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.430447+0000 mon.a (mon.0) 412 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.430600+0000 mon.a (mon.0) 413 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-10T17:12:46.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.430850+0000 mon.a (mon.0) 414 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-10T17:12:46.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.431067+0000 mon.a (mon.0) 415 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-10T17:12:46.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.563741+0000 mon.a (mon.0) 416 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.565409+0000 mon.a (mon.0) 417 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.565545+0000 mon.a (mon.0) 418 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:45 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.565829+0000 mon.a (mon.0) 419 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.021699+0000 mon.a (mon.0) 383 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.021883+0000 mon.a (mon.0) 384 : cluster 1 osd.2 failed (root=default,host=smithi116) (connection refused reported by osd.5) 2023-07-10T17:12:46.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.022611+0000 mon.a (mon.0) 385 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.023419+0000 mon.a (mon.0) 386 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.024036+0000 mon.a (mon.0) 387 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-10T17:12:46.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.024436+0000 mon.a (mon.0) 388 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.024599+0000 mon.a (mon.0) 389 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.024850+0000 mon.a (mon.0) 390 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.025021+0000 mon.a (mon.0) 391 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-10T17:12:46.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.025315+0000 mon.a (mon.0) 392 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.025416+0000 mon.a (mon.0) 393 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.025687+0000 mon.a (mon.0) 394 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-10T17:12:46.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.025901+0000 mon.a (mon.0) 395 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-10T17:12:46.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.026264+0000 mon.a (mon.0) 396 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-10T17:12:46.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.026580+0000 mon.a (mon.0) 397 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-10T17:12:46.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.359977+0000 mon.a (mon.0) 398 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.360140+0000 mon.a (mon.0) 399 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.360527+0000 mon.a (mon.0) 400 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.361050+0000 mon.a (mon.0) 401 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.427246+0000 mon.a (mon.0) 402 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.427730+0000 mon.a (mon.0) 403 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.428372+0000 mon.a (mon.0) 404 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-10T17:12:46.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.428794+0000 mon.a (mon.0) 405 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.429093+0000 mon.a (mon.0) 406 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:46.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.429308+0000 mon.a (mon.0) 407 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.429601+0000 mon.a (mon.0) 408 : cluster 0 osd.2 reported immediately failed by osd.6 2023-07-10T17:12:46.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.429843+0000 mon.a (mon.0) 409 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.430127+0000 mon.a (mon.0) 410 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.430234+0000 mon.a (mon.0) 411 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-10T17:12:46.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.430447+0000 mon.a (mon.0) 412 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.430600+0000 mon.a (mon.0) 413 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-10T17:12:46.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.430850+0000 mon.a (mon.0) 414 : cluster 0 osd.2 reported immediately failed by osd.1 2023-07-10T17:12:46.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.431067+0000 mon.a (mon.0) 415 : cluster 0 osd.2 reported immediately failed by osd.0 2023-07-10T17:12:46.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.563741+0000 mon.a (mon.0) 416 : cluster 0 osd.2 reported immediately failed by osd.3 2023-07-10T17:12:46.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.565409+0000 mon.a (mon.0) 417 : cluster 0 osd.2 reported immediately failed by osd.4 2023-07-10T17:12:46.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.565545+0000 mon.a (mon.0) 418 : cluster 0 osd.2 reported immediately failed by osd.7 2023-07-10T17:12:46.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:45 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.565829+0000 mon.a (mon.0) 419 : cluster 0 osd.2 reported immediately failed by osd.5 2023-07-10T17:12:47.366 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:46 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.383715+0000 mgr.y (mgr.25390) 186 : cluster 0 pgmap v111: 161 pgs: 161 active+clean; 604 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2023-07-10T17:12:47.366 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:46 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.933197+0000 mon.a (mon.0) 420 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:12:47.367 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:46 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.945799+0000 osd.3 (osd.3) 14 : cluster 3 failed to encode map e88 with expected crc 2023-07-10T17:12:47.367 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:46 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.948237+0000 osd.6 (osd.6) 16 : cluster 3 failed to encode map e88 with expected crc 2023-07-10T17:12:47.367 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:46 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.953641+0000 mon.a (mon.0) 421 : cluster 0 osdmap e88: 8 total, 7 up, 8 in 2023-07-10T17:12:47.367 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:46 smithi116 bash[65980]: cluster 2023-07-10T17:12:45.955955+0000 osd.4 (osd.4) 12 : cluster 3 failed to encode map e88 with expected crc 2023-07-10T17:12:47.367 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:46 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.383715+0000 mgr.y (mgr.25390) 186 : cluster 0 pgmap v111: 161 pgs: 161 active+clean; 604 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2023-07-10T17:12:47.367 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:46 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.933197+0000 mon.a (mon.0) 420 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:12:47.367 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:46 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.945799+0000 osd.3 (osd.3) 14 : cluster 3 failed to encode map e88 with expected crc 2023-07-10T17:12:47.368 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:46 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.948237+0000 osd.6 (osd.6) 16 : cluster 3 failed to encode map e88 with expected crc 2023-07-10T17:12:47.368 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:46 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.953641+0000 mon.a (mon.0) 421 : cluster 0 osdmap e88: 8 total, 7 up, 8 in 2023-07-10T17:12:47.368 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:46 smithi116 bash[63900]: cluster 2023-07-10T17:12:45.955955+0000 osd.4 (osd.4) 12 : cluster 3 failed to encode map e88 with expected crc 2023-07-10T17:12:47.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:46 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.383715+0000 mgr.y (mgr.25390) 186 : cluster 0 pgmap v111: 161 pgs: 161 active+clean; 604 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2023-07-10T17:12:47.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:46 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.933197+0000 mon.a (mon.0) 420 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:12:47.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:46 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.945799+0000 osd.3 (osd.3) 14 : cluster 3 failed to encode map e88 with expected crc 2023-07-10T17:12:47.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:46 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.948237+0000 osd.6 (osd.6) 16 : cluster 3 failed to encode map e88 with expected crc 2023-07-10T17:12:47.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:46 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.953641+0000 mon.a (mon.0) 421 : cluster 0 osdmap e88: 8 total, 7 up, 8 in 2023-07-10T17:12:47.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:46 smithi192 bash[50362]: cluster 2023-07-10T17:12:45.955955+0000 osd.4 (osd.4) 12 : cluster 3 failed to encode map e88 with expected crc 2023-07-10T17:12:48.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:47 smithi192 bash[50362]: cluster 2023-07-10T17:12:46.953735+0000 osd.3 (osd.3) 15 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:47 smithi192 bash[50362]: cluster 2023-07-10T17:12:46.954766+0000 osd.6 (osd.6) 17 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:47 smithi192 bash[50362]: cluster 2023-07-10T17:12:46.957175+0000 mon.a (mon.0) 422 : cluster 0 osdmap e89: 8 total, 7 up, 8 in 2023-07-10T17:12:48.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:47 smithi192 bash[50362]: cluster 2023-07-10T17:12:46.957535+0000 osd.3 (osd.3) 16 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:47 smithi192 bash[50362]: cluster 2023-07-10T17:12:46.958570+0000 osd.4 (osd.4) 13 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:47 smithi192 bash[50362]: cluster 2023-07-10T17:12:46.958603+0000 osd.7 (osd.7) 17 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:47 smithi192 bash[50362]: cluster 2023-07-10T17:12:46.958693+0000 osd.5 (osd.5) 13 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:47 smithi192 bash[50362]: cluster 2023-07-10T17:12:46.958809+0000 osd.6 (osd.6) 18 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:47 smithi192 bash[50362]: cluster 2023-07-10T17:12:46.962554+0000 osd.4 (osd.4) 14 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:47 smithi192 bash[50362]: cluster 2023-07-10T17:12:46.962556+0000 osd.5 (osd.5) 14 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[63900]: cluster 2023-07-10T17:12:46.953735+0000 osd.3 (osd.3) 15 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[63900]: cluster 2023-07-10T17:12:46.954766+0000 osd.6 (osd.6) 17 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[63900]: cluster 2023-07-10T17:12:46.957175+0000 mon.a (mon.0) 422 : cluster 0 osdmap e89: 8 total, 7 up, 8 in 2023-07-10T17:12:48.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[63900]: cluster 2023-07-10T17:12:46.957535+0000 osd.3 (osd.3) 16 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[63900]: cluster 2023-07-10T17:12:46.958570+0000 osd.4 (osd.4) 13 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[63900]: cluster 2023-07-10T17:12:46.958603+0000 osd.7 (osd.7) 17 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[63900]: cluster 2023-07-10T17:12:46.958693+0000 osd.5 (osd.5) 13 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[63900]: cluster 2023-07-10T17:12:46.958809+0000 osd.6 (osd.6) 18 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[63900]: cluster 2023-07-10T17:12:46.962554+0000 osd.4 (osd.4) 14 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[63900]: cluster 2023-07-10T17:12:46.962556+0000 osd.5 (osd.5) 14 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[65980]: cluster 2023-07-10T17:12:46.953735+0000 osd.3 (osd.3) 15 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[65980]: cluster 2023-07-10T17:12:46.954766+0000 osd.6 (osd.6) 17 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[65980]: cluster 2023-07-10T17:12:46.957175+0000 mon.a (mon.0) 422 : cluster 0 osdmap e89: 8 total, 7 up, 8 in 2023-07-10T17:12:48.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[65980]: cluster 2023-07-10T17:12:46.957535+0000 osd.3 (osd.3) 16 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[65980]: cluster 2023-07-10T17:12:46.958570+0000 osd.4 (osd.4) 13 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[65980]: cluster 2023-07-10T17:12:46.958603+0000 osd.7 (osd.7) 17 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[65980]: cluster 2023-07-10T17:12:46.958693+0000 osd.5 (osd.5) 13 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[65980]: cluster 2023-07-10T17:12:46.958809+0000 osd.6 (osd.6) 18 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[65980]: cluster 2023-07-10T17:12:46.962554+0000 osd.4 (osd.4) 14 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:47 smithi116 bash[65980]: cluster 2023-07-10T17:12:46.962556+0000 osd.5 (osd.5) 14 : cluster 3 failed to encode map e89 with expected crc 2023-07-10T17:12:48.396 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:48 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.2.service: Succeeded. 2023-07-10T17:12:48.396 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:48 smithi116 systemd[1]: Stopped Ceph osd.2 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:12:48.396 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:48 smithi116 systemd[1]: Started Ceph osd.2 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:12:49.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:48 smithi192 bash[50362]: cluster 2023-07-10T17:12:47.384984+0000 mgr.y (mgr.25390) 187 : cluster 0 pgmap v114: 161 pgs: 7 active+undersized+degraded, 7 peering, 2 stale+active+clean, 25 active+undersized, 120 active+clean; 604 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 0 B/s wr, 171 op/s; 47/999 objects degraded (4.705%) 2023-07-10T17:12:49.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:48 smithi192 bash[50362]: cluster 2023-07-10T17:12:47.951883+0000 mon.a (mon.0) 423 : cluster 3 Health check failed: Degraded data redundancy: 47/999 objects degraded (4.705%), 7 pgs degraded (PG_DEGRADED) 2023-07-10T17:12:49.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:48 smithi192 bash[50362]: audit 2023-07-10T17:12:48.427417+0000 mon.a (mon.0) 424 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:49.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:48 smithi192 bash[50362]: audit 2023-07-10T17:12:48.437354+0000 mon.a (mon.0) 425 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:49.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:48 smithi116 bash[63900]: cluster 2023-07-10T17:12:47.384984+0000 mgr.y (mgr.25390) 187 : cluster 0 pgmap v114: 161 pgs: 7 active+undersized+degraded, 7 peering, 2 stale+active+clean, 25 active+undersized, 120 active+clean; 604 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 0 B/s wr, 171 op/s; 47/999 objects degraded (4.705%) 2023-07-10T17:12:49.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:48 smithi116 bash[63900]: cluster 2023-07-10T17:12:47.951883+0000 mon.a (mon.0) 423 : cluster 3 Health check failed: Degraded data redundancy: 47/999 objects degraded (4.705%), 7 pgs degraded (PG_DEGRADED) 2023-07-10T17:12:49.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:48 smithi116 bash[63900]: audit 2023-07-10T17:12:48.427417+0000 mon.a (mon.0) 424 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:49.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:48 smithi116 bash[63900]: audit 2023-07-10T17:12:48.437354+0000 mon.a (mon.0) 425 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:49.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:48 smithi116 bash[65980]: cluster 2023-07-10T17:12:47.384984+0000 mgr.y (mgr.25390) 187 : cluster 0 pgmap v114: 161 pgs: 7 active+undersized+degraded, 7 peering, 2 stale+active+clean, 25 active+undersized, 120 active+clean; 604 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 0 B/s wr, 171 op/s; 47/999 objects degraded (4.705%) 2023-07-10T17:12:49.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:48 smithi116 bash[65980]: cluster 2023-07-10T17:12:47.951883+0000 mon.a (mon.0) 423 : cluster 3 Health check failed: Degraded data redundancy: 47/999 objects degraded (4.705%), 7 pgs degraded (PG_DEGRADED) 2023-07-10T17:12:49.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:48 smithi116 bash[65980]: audit 2023-07-10T17:12:48.427417+0000 mon.a (mon.0) 424 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:49.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:48 smithi116 bash[65980]: audit 2023-07-10T17:12:48.437354+0000 mon.a (mon.0) 425 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:50.644 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:50 smithi116 bash[74875]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-07-10T17:12:50.644 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:50 smithi116 bash[74875]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2023-07-10T17:12:50.645 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:50 smithi116 bash[74875]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2023-07-10T17:12:50.645 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:50 smithi116 bash[74875]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2023-07-10T17:12:50.645 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:50 smithi116 bash[74875]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2023-07-10T17:12:50.645 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:50 smithi116 bash[74875]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-07-10T17:12:50.645 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:50 smithi116 bash[74875]: --> ceph-volume raw activate successful for osd ID: 2 2023-07-10T17:12:51.290 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:50 smithi116 bash[63900]: cluster 2023-07-10T17:12:49.385720+0000 mgr.y (mgr.25390) 188 : cluster 0 pgmap v115: 161 pgs: 7 active+undersized+degraded, 7 peering, 2 stale+active+clean, 25 active+undersized, 120 active+clean; 604 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 160 op/s; 47/999 objects degraded (4.705%) 2023-07-10T17:12:51.290 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:50 smithi116 bash[65980]: cluster 2023-07-10T17:12:49.385720+0000 mgr.y (mgr.25390) 188 : cluster 0 pgmap v115: 161 pgs: 7 active+undersized+degraded, 7 peering, 2 stale+active+clean, 25 active+undersized, 120 active+clean; 604 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 160 op/s; 47/999 objects degraded (4.705%) 2023-07-10T17:12:51.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:50 smithi192 bash[50362]: cluster 2023-07-10T17:12:49.385720+0000 mgr.y (mgr.25390) 188 : cluster 0 pgmap v115: 161 pgs: 7 active+undersized+degraded, 7 peering, 2 stale+active+clean, 25 active+undersized, 120 active+clean; 604 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 160 op/s; 47/999 objects degraded (4.705%) 2023-07-10T17:12:52.644 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:52 smithi116 bash[75175]: debug 2023-07-10T17:12:52.426+0000 7f2946091700 1 -- 172.21.15.116:0/1839246269 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558ee5bf64e0 con 0x558ee5c50c00 2023-07-10T17:12:53.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:52 smithi192 bash[50362]: cluster 2023-07-10T17:12:51.386901+0000 mgr.y (mgr.25390) 189 : cluster 0 pgmap v116: 161 pgs: 10 active+undersized+degraded, 7 peering, 30 active+undersized, 114 active+clean; 604 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s; 74/999 objects degraded (7.407%) 2023-07-10T17:12:53.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:52 smithi116 bash[65980]: cluster 2023-07-10T17:12:51.386901+0000 mgr.y (mgr.25390) 189 : cluster 0 pgmap v116: 161 pgs: 10 active+undersized+degraded, 7 peering, 30 active+undersized, 114 active+clean; 604 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s; 74/999 objects degraded (7.407%) 2023-07-10T17:12:53.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:52 smithi116 bash[63900]: cluster 2023-07-10T17:12:51.386901+0000 mgr.y (mgr.25390) 189 : cluster 0 pgmap v116: 161 pgs: 10 active+undersized+degraded, 7 peering, 30 active+undersized, 114 active+clean; 604 KiB data, 145 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s; 74/999 objects degraded (7.407%) 2023-07-10T17:12:53.394 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:52 smithi116 bash[75175]: debug 2023-07-10T17:12:52.966+0000 7f294f216640 -1 Falling back to public interface 2023-07-10T17:12:54.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:54 smithi192 bash[50362]: cluster 2023-07-10T17:12:53.974874+0000 mon.a (mon.0) 426 : cluster 3 Health check update: Degraded data redundancy: 85/999 objects degraded (8.509%), 11 pgs degraded (PG_DEGRADED) 2023-07-10T17:12:54.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:54 smithi116 bash[63900]: cluster 2023-07-10T17:12:53.974874+0000 mon.a (mon.0) 426 : cluster 3 Health check update: Degraded data redundancy: 85/999 objects degraded (8.509%), 11 pgs degraded (PG_DEGRADED) 2023-07-10T17:12:54.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:54 smithi116 bash[65980]: cluster 2023-07-10T17:12:53.974874+0000 mon.a (mon.0) 426 : cluster 3 Health check update: Degraded data redundancy: 85/999 objects degraded (8.509%), 11 pgs degraded (PG_DEGRADED) 2023-07-10T17:12:55.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:55 smithi192 bash[50362]: cluster 2023-07-10T17:12:53.388732+0000 mgr.y (mgr.25390) 190 : cluster 0 pgmap v117: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s; 85/999 objects degraded (8.509%) 2023-07-10T17:12:55.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:55 smithi116 bash[65980]: cluster 2023-07-10T17:12:53.388732+0000 mgr.y (mgr.25390) 190 : cluster 0 pgmap v117: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s; 85/999 objects degraded (8.509%) 2023-07-10T17:12:55.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:55 smithi116 bash[63900]: cluster 2023-07-10T17:12:53.388732+0000 mgr.y (mgr.25390) 190 : cluster 0 pgmap v117: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s; 85/999 objects degraded (8.509%) 2023-07-10T17:12:56.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:56 smithi192 bash[50362]: audit 2023-07-10T17:12:55.981679+0000 mon.c (mon.1) 19 : audit 1 from='osd.2 [v2:172.21.15.116:6818/322392750,v1:172.21.15.116:6819/322392750]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-10T17:12:56.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:56 smithi192 bash[50362]: audit 2023-07-10T17:12:55.982369+0000 mon.a (mon.0) 427 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-10T17:12:56.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:56 smithi116 bash[63900]: audit 2023-07-10T17:12:55.981679+0000 mon.c (mon.1) 19 : audit 1 from='osd.2 [v2:172.21.15.116:6818/322392750,v1:172.21.15.116:6819/322392750]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-10T17:12:56.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:56 smithi116 bash[63900]: audit 2023-07-10T17:12:55.982369+0000 mon.a (mon.0) 427 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-10T17:12:56.394 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:55 smithi116 bash[75175]: debug 2023-07-10T17:12:55.974+0000 7f294f216640 -1 osd.2 87 log_to_monitors true 2023-07-10T17:12:56.394 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:12:56 smithi116 bash[75175]: debug 2023-07-10T17:12:56.090+0000 7f293e67f700 -1 osd.2 87 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-10T17:12:56.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:56 smithi116 bash[65980]: audit 2023-07-10T17:12:55.981679+0000 mon.c (mon.1) 19 : audit 1 from='osd.2 [v2:172.21.15.116:6818/322392750,v1:172.21.15.116:6819/322392750]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-10T17:12:56.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:56 smithi116 bash[65980]: audit 2023-07-10T17:12:55.982369+0000 mon.a (mon.0) 427 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-10T17:12:57.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:57 smithi192 bash[50362]: cluster 2023-07-10T17:12:55.389508+0000 mgr.y (mgr.25390) 191 : cluster 0 pgmap v118: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s; 85/999 objects degraded (8.509%) 2023-07-10T17:12:57.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:57 smithi192 bash[50362]: cluster 2023-07-10T17:12:56.041785+0000 osd.3 (osd.3) 17 : cluster 3 failed to encode map e90 with expected crc 2023-07-10T17:12:57.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:57 smithi192 bash[50362]: cluster 2023-07-10T17:12:56.043530+0000 osd.6 (osd.6) 19 : cluster 3 failed to encode map e90 with expected crc 2023-07-10T17:12:57.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:57 smithi192 bash[50362]: audit 2023-07-10T17:12:56.045022+0000 mon.a (mon.0) 428 : audit 1 from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-10T17:12:57.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:57 smithi192 bash[50362]: cluster 2023-07-10T17:12:56.045091+0000 mon.a (mon.0) 429 : cluster 0 osdmap e90: 8 total, 7 up, 8 in 2023-07-10T17:12:57.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:57 smithi192 bash[50362]: audit 2023-07-10T17:12:56.048069+0000 mon.c (mon.1) 20 : audit 1 from='osd.2 [v2:172.21.15.116:6818/322392750,v1:172.21.15.116:6819/322392750]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:12:57.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:57 smithi192 bash[50362]: audit 2023-07-10T17:12:56.048559+0000 mon.a (mon.0) 430 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:12:57.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:57 smithi192 bash[50362]: audit 2023-07-10T17:12:56.351692+0000 mon.a (mon.0) 431 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:57.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:57 smithi192 bash[50362]: audit 2023-07-10T17:12:56.364722+0000 mon.a (mon.0) 432 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:57.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[65980]: cluster 2023-07-10T17:12:55.389508+0000 mgr.y (mgr.25390) 191 : cluster 0 pgmap v118: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s; 85/999 objects degraded (8.509%) 2023-07-10T17:12:57.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[65980]: cluster 2023-07-10T17:12:56.041785+0000 osd.3 (osd.3) 17 : cluster 3 failed to encode map e90 with expected crc 2023-07-10T17:12:57.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[65980]: cluster 2023-07-10T17:12:56.043530+0000 osd.6 (osd.6) 19 : cluster 3 failed to encode map e90 with expected crc 2023-07-10T17:12:57.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[65980]: audit 2023-07-10T17:12:56.045022+0000 mon.a (mon.0) 428 : audit 1 from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-10T17:12:57.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[65980]: cluster 2023-07-10T17:12:56.045091+0000 mon.a (mon.0) 429 : cluster 0 osdmap e90: 8 total, 7 up, 8 in 2023-07-10T17:12:57.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[65980]: audit 2023-07-10T17:12:56.048069+0000 mon.c (mon.1) 20 : audit 1 from='osd.2 [v2:172.21.15.116:6818/322392750,v1:172.21.15.116:6819/322392750]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:12:57.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[65980]: audit 2023-07-10T17:12:56.048559+0000 mon.a (mon.0) 430 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:12:57.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[65980]: audit 2023-07-10T17:12:56.351692+0000 mon.a (mon.0) 431 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:57.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[65980]: audit 2023-07-10T17:12:56.364722+0000 mon.a (mon.0) 432 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:57.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[63900]: cluster 2023-07-10T17:12:55.389508+0000 mgr.y (mgr.25390) 191 : cluster 0 pgmap v118: 161 pgs: 11 active+undersized+degraded, 36 active+undersized, 114 active+clean; 604 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s; 85/999 objects degraded (8.509%) 2023-07-10T17:12:57.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[63900]: cluster 2023-07-10T17:12:56.041785+0000 osd.3 (osd.3) 17 : cluster 3 failed to encode map e90 with expected crc 2023-07-10T17:12:57.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[63900]: cluster 2023-07-10T17:12:56.043530+0000 osd.6 (osd.6) 19 : cluster 3 failed to encode map e90 with expected crc 2023-07-10T17:12:57.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[63900]: audit 2023-07-10T17:12:56.045022+0000 mon.a (mon.0) 428 : audit 1 from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-10T17:12:57.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[63900]: cluster 2023-07-10T17:12:56.045091+0000 mon.a (mon.0) 429 : cluster 0 osdmap e90: 8 total, 7 up, 8 in 2023-07-10T17:12:57.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[63900]: audit 2023-07-10T17:12:56.048069+0000 mon.c (mon.1) 20 : audit 1 from='osd.2 [v2:172.21.15.116:6818/322392750,v1:172.21.15.116:6819/322392750]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:12:57.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[63900]: audit 2023-07-10T17:12:56.048559+0000 mon.a (mon.0) 430 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:12:57.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[63900]: audit 2023-07-10T17:12:56.351692+0000 mon.a (mon.0) 431 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:57.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:57 smithi116 bash[63900]: audit 2023-07-10T17:12:56.364722+0000 mon.a (mon.0) 432 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:58.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:58 smithi192 bash[50362]: cluster 2023-07-10T17:12:57.041243+0000 mon.a (mon.0) 433 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:12:58.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:58 smithi192 bash[50362]: cluster 2023-07-10T17:12:57.046636+0000 osd.3 (osd.3) 18 : cluster 3 failed to encode map e91 with expected crc 2023-07-10T17:12:58.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:58 smithi192 bash[50362]: cluster 2023-07-10T17:12:57.048080+0000 osd.6 (osd.6) 20 : cluster 3 failed to encode map e91 with expected crc 2023-07-10T17:12:58.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:58 smithi192 bash[50362]: cluster 2023-07-10T17:12:57.050896+0000 mon.a (mon.0) 434 : cluster 1 osd.2 [v2:172.21.15.116:6818/322392750,v1:172.21.15.116:6819/322392750] boot 2023-07-10T17:12:58.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:58 smithi192 bash[50362]: cluster 2023-07-10T17:12:57.051053+0000 mon.a (mon.0) 435 : cluster 0 osdmap e91: 8 total, 8 up, 8 in 2023-07-10T17:12:58.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:58 smithi192 bash[50362]: audit 2023-07-10T17:12:57.052240+0000 mon.a (mon.0) 436 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T17:12:58.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:58 smithi192 bash[50362]: cluster 2023-07-10T17:12:57.055254+0000 osd.4 (osd.4) 15 : cluster 3 failed to encode map e91 with expected crc 2023-07-10T17:12:58.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:58 smithi192 bash[50362]: audit 2023-07-10T17:12:57.159281+0000 mon.a (mon.0) 437 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:58.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:58 smithi192 bash[50362]: audit 2023-07-10T17:12:57.168954+0000 mon.a (mon.0) 438 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:58.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[63900]: cluster 2023-07-10T17:12:57.041243+0000 mon.a (mon.0) 433 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:12:58.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[63900]: cluster 2023-07-10T17:12:57.046636+0000 osd.3 (osd.3) 18 : cluster 3 failed to encode map e91 with expected crc 2023-07-10T17:12:58.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[63900]: cluster 2023-07-10T17:12:57.048080+0000 osd.6 (osd.6) 20 : cluster 3 failed to encode map e91 with expected crc 2023-07-10T17:12:58.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[63900]: cluster 2023-07-10T17:12:57.050896+0000 mon.a (mon.0) 434 : cluster 1 osd.2 [v2:172.21.15.116:6818/322392750,v1:172.21.15.116:6819/322392750] boot 2023-07-10T17:12:58.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[63900]: cluster 2023-07-10T17:12:57.051053+0000 mon.a (mon.0) 435 : cluster 0 osdmap e91: 8 total, 8 up, 8 in 2023-07-10T17:12:58.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[63900]: audit 2023-07-10T17:12:57.052240+0000 mon.a (mon.0) 436 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T17:12:58.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[63900]: cluster 2023-07-10T17:12:57.055254+0000 osd.4 (osd.4) 15 : cluster 3 failed to encode map e91 with expected crc 2023-07-10T17:12:58.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[63900]: audit 2023-07-10T17:12:57.159281+0000 mon.a (mon.0) 437 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:58.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[63900]: audit 2023-07-10T17:12:57.168954+0000 mon.a (mon.0) 438 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:58.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[65980]: cluster 2023-07-10T17:12:57.041243+0000 mon.a (mon.0) 433 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:12:58.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[65980]: cluster 2023-07-10T17:12:57.046636+0000 osd.3 (osd.3) 18 : cluster 3 failed to encode map e91 with expected crc 2023-07-10T17:12:58.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[65980]: cluster 2023-07-10T17:12:57.048080+0000 osd.6 (osd.6) 20 : cluster 3 failed to encode map e91 with expected crc 2023-07-10T17:12:58.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[65980]: cluster 2023-07-10T17:12:57.050896+0000 mon.a (mon.0) 434 : cluster 1 osd.2 [v2:172.21.15.116:6818/322392750,v1:172.21.15.116:6819/322392750] boot 2023-07-10T17:12:58.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[65980]: cluster 2023-07-10T17:12:57.051053+0000 mon.a (mon.0) 435 : cluster 0 osdmap e91: 8 total, 8 up, 8 in 2023-07-10T17:12:58.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[65980]: audit 2023-07-10T17:12:57.052240+0000 mon.a (mon.0) 436 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-10T17:12:58.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[65980]: cluster 2023-07-10T17:12:57.055254+0000 osd.4 (osd.4) 15 : cluster 3 failed to encode map e91 with expected crc 2023-07-10T17:12:58.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[65980]: audit 2023-07-10T17:12:57.159281+0000 mon.a (mon.0) 437 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:58.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:58 smithi116 bash[65980]: audit 2023-07-10T17:12:57.168954+0000 mon.a (mon.0) 438 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:12:59.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:59 smithi192 bash[50362]: cluster 2023-07-10T17:12:57.391312+0000 mgr.y (mgr.25390) 192 : cluster 0 pgmap v121: 161 pgs: 24 peering, 4 active+undersized+degraded, 19 active+undersized, 114 active+clean; 604 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 0 B/s wr, 178 op/s; 38/999 objects degraded (3.804%) 2023-07-10T17:12:59.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:59 smithi192 bash[50362]: cluster 2023-07-10T17:12:58.070048+0000 osd.3 (osd.3) 19 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:59 smithi192 bash[50362]: cluster 2023-07-10T17:12:58.070909+0000 osd.6 (osd.6) 21 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:59 smithi192 bash[50362]: cluster 2023-07-10T17:12:58.072402+0000 mon.a (mon.0) 439 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2023-07-10T17:12:59.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:59 smithi192 bash[50362]: cluster 2023-07-10T17:12:58.073674+0000 osd.3 (osd.3) 20 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:59 smithi192 bash[50362]: cluster 2023-07-10T17:12:58.073833+0000 osd.4 (osd.4) 16 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:59 smithi192 bash[50362]: cluster 2023-07-10T17:12:58.073920+0000 osd.5 (osd.5) 15 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:59 smithi192 bash[50362]: cluster 2023-07-10T17:12:58.074088+0000 osd.6 (osd.6) 22 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:12:59 smithi192 bash[50362]: cluster 2023-07-10T17:12:58.083084+0000 osd.5 (osd.5) 16 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[63900]: cluster 2023-07-10T17:12:57.391312+0000 mgr.y (mgr.25390) 192 : cluster 0 pgmap v121: 161 pgs: 24 peering, 4 active+undersized+degraded, 19 active+undersized, 114 active+clean; 604 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 0 B/s wr, 178 op/s; 38/999 objects degraded (3.804%) 2023-07-10T17:12:59.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[63900]: cluster 2023-07-10T17:12:58.070048+0000 osd.3 (osd.3) 19 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[63900]: cluster 2023-07-10T17:12:58.070909+0000 osd.6 (osd.6) 21 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[63900]: cluster 2023-07-10T17:12:58.072402+0000 mon.a (mon.0) 439 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2023-07-10T17:12:59.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[63900]: cluster 2023-07-10T17:12:58.073674+0000 osd.3 (osd.3) 20 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[63900]: cluster 2023-07-10T17:12:58.073833+0000 osd.4 (osd.4) 16 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[63900]: cluster 2023-07-10T17:12:58.073920+0000 osd.5 (osd.5) 15 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[63900]: cluster 2023-07-10T17:12:58.074088+0000 osd.6 (osd.6) 22 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[63900]: cluster 2023-07-10T17:12:58.083084+0000 osd.5 (osd.5) 16 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[65980]: cluster 2023-07-10T17:12:57.391312+0000 mgr.y (mgr.25390) 192 : cluster 0 pgmap v121: 161 pgs: 24 peering, 4 active+undersized+degraded, 19 active+undersized, 114 active+clean; 604 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 0 B/s wr, 178 op/s; 38/999 objects degraded (3.804%) 2023-07-10T17:12:59.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[65980]: cluster 2023-07-10T17:12:58.070048+0000 osd.3 (osd.3) 19 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[65980]: cluster 2023-07-10T17:12:58.070909+0000 osd.6 (osd.6) 21 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[65980]: cluster 2023-07-10T17:12:58.072402+0000 mon.a (mon.0) 439 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2023-07-10T17:12:59.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[65980]: cluster 2023-07-10T17:12:58.073674+0000 osd.3 (osd.3) 20 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[65980]: cluster 2023-07-10T17:12:58.073833+0000 osd.4 (osd.4) 16 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[65980]: cluster 2023-07-10T17:12:58.073920+0000 osd.5 (osd.5) 15 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[65980]: cluster 2023-07-10T17:12:58.074088+0000 osd.6 (osd.6) 22 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:12:59.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:12:59 smithi116 bash[65980]: cluster 2023-07-10T17:12:58.083084+0000 osd.5 (osd.5) 16 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:13:00.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:00 smithi192 bash[50362]: cluster 2023-07-10T17:12:58.074085+0000 osd.7 (osd.7) 18 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:13:00.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:00 smithi116 bash[63900]: cluster 2023-07-10T17:12:58.074085+0000 osd.7 (osd.7) 18 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:13:00.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:00 smithi116 bash[65980]: cluster 2023-07-10T17:12:58.074085+0000 osd.7 (osd.7) 18 : cluster 3 failed to encode map e92 with expected crc 2023-07-10T17:13:01.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:01 smithi192 bash[50362]: cluster 2023-07-10T17:12:59.392204+0000 mgr.y (mgr.25390) 193 : cluster 0 pgmap v123: 161 pgs: 24 peering, 4 active+undersized+degraded, 19 active+undersized, 114 active+clean; 604 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s; 38/999 objects degraded (3.804%) 2023-07-10T17:13:01.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:01 smithi116 bash[63900]: cluster 2023-07-10T17:12:59.392204+0000 mgr.y (mgr.25390) 193 : cluster 0 pgmap v123: 161 pgs: 24 peering, 4 active+undersized+degraded, 19 active+undersized, 114 active+clean; 604 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s; 38/999 objects degraded (3.804%) 2023-07-10T17:13:01.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:01 smithi116 bash[65980]: cluster 2023-07-10T17:12:59.392204+0000 mgr.y (mgr.25390) 193 : cluster 0 pgmap v123: 161 pgs: 24 peering, 4 active+undersized+degraded, 19 active+undersized, 114 active+clean; 604 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s; 38/999 objects degraded (3.804%) 2023-07-10T17:13:02.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:02 smithi116 bash[65980]: cluster 2023-07-10T17:13:02.080285+0000 mon.a (mon.0) 440 : cluster 3 Health check update: Degraded data redundancy: 11/999 objects degraded (1.101%), 1 pg degraded (PG_DEGRADED) 2023-07-10T17:13:02.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:02 smithi116 bash[63900]: cluster 2023-07-10T17:13:02.080285+0000 mon.a (mon.0) 440 : cluster 3 Health check update: Degraded data redundancy: 11/999 objects degraded (1.101%), 1 pg degraded (PG_DEGRADED) 2023-07-10T17:13:02.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:02 smithi192 bash[50362]: cluster 2023-07-10T17:13:02.080285+0000 mon.a (mon.0) 440 : cluster 3 Health check update: Degraded data redundancy: 11/999 objects degraded (1.101%), 1 pg degraded (PG_DEGRADED) 2023-07-10T17:13:03.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:03 smithi116 bash[65980]: cluster 2023-07-10T17:13:01.393275+0000 mgr.y (mgr.25390) 194 : cluster 0 pgmap v124: 161 pgs: 24 peering, 1 active+undersized+degraded, 4 active+undersized, 132 active+clean; 604 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s; 11/999 objects degraded (1.101%); 0 B/s, 1 objects/s recovering 2023-07-10T17:13:03.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:03 smithi116 bash[63900]: cluster 2023-07-10T17:13:01.393275+0000 mgr.y (mgr.25390) 194 : cluster 0 pgmap v124: 161 pgs: 24 peering, 1 active+undersized+degraded, 4 active+undersized, 132 active+clean; 604 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s; 11/999 objects degraded (1.101%); 0 B/s, 1 objects/s recovering 2023-07-10T17:13:03.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:03 smithi192 bash[50362]: cluster 2023-07-10T17:13:01.393275+0000 mgr.y (mgr.25390) 194 : cluster 0 pgmap v124: 161 pgs: 24 peering, 1 active+undersized+degraded, 4 active+undersized, 132 active+clean; 604 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s; 11/999 objects degraded (1.101%); 0 B/s, 1 objects/s recovering 2023-07-10T17:13:04.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:04 smithi192 bash[50362]: cluster 2023-07-10T17:13:04.137180+0000 mon.a (mon.0) 441 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/999 objects degraded (1.101%), 1 pg degraded) 2023-07-10T17:13:04.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:04 smithi116 bash[63900]: cluster 2023-07-10T17:13:04.137180+0000 mon.a (mon.0) 441 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/999 objects degraded (1.101%), 1 pg degraded) 2023-07-10T17:13:04.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:04 smithi116 bash[65980]: cluster 2023-07-10T17:13:04.137180+0000 mon.a (mon.0) 441 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/999 objects degraded (1.101%), 1 pg degraded) 2023-07-10T17:13:05.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:05 smithi192 bash[50362]: cluster 2023-07-10T17:13:03.394803+0000 mgr.y (mgr.25390) 195 : cluster 0 pgmap v125: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s; 0 B/s, 5 objects/s recovering 2023-07-10T17:13:05.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:05 smithi116 bash[63900]: cluster 2023-07-10T17:13:03.394803+0000 mgr.y (mgr.25390) 195 : cluster 0 pgmap v125: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s; 0 B/s, 5 objects/s recovering 2023-07-10T17:13:05.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:05 smithi116 bash[65980]: cluster 2023-07-10T17:13:03.394803+0000 mgr.y (mgr.25390) 195 : cluster 0 pgmap v125: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s; 0 B/s, 5 objects/s recovering 2023-07-10T17:13:07.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:07 smithi192 bash[50362]: cluster 2023-07-10T17:13:05.395581+0000 mgr.y (mgr.25390) 196 : cluster 0 pgmap v126: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 53 op/s; 0 B/s, 4 objects/s recovering 2023-07-10T17:13:07.643 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:07 smithi116 bash[63900]: cluster 2023-07-10T17:13:05.395581+0000 mgr.y (mgr.25390) 196 : cluster 0 pgmap v126: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 53 op/s; 0 B/s, 4 objects/s recovering 2023-07-10T17:13:07.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:07 smithi116 bash[65980]: cluster 2023-07-10T17:13:05.395581+0000 mgr.y (mgr.25390) 196 : cluster 0 pgmap v126: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 53 op/s; 0 B/s, 4 objects/s recovering 2023-07-10T17:13:08.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:08 smithi192 bash[50362]: audit 2023-07-10T17:13:07.374411+0000 mon.a (mon.0) 442 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:08.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:08 smithi192 bash[50362]: audit 2023-07-10T17:13:07.386009+0000 mon.a (mon.0) 443 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:08.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:08 smithi192 bash[50362]: audit 2023-07-10T17:13:07.388344+0000 mon.a (mon.0) 444 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:13:08.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:08 smithi192 bash[50362]: audit 2023-07-10T17:13:07.390098+0000 mon.a (mon.0) 445 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:13:08.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:08 smithi192 bash[50362]: cephadm 2023-07-10T17:13:07.391578+0000 mgr.y (mgr.25390) 197 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:13:08.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:08 smithi192 bash[50362]: cluster 2023-07-10T17:13:07.393808+0000 mgr.y (mgr.25390) 198 : cluster 0 pgmap v127: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 3 objects/s recovering 2023-07-10T17:13:08.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:08 smithi192 bash[50362]: cluster 2023-07-10T17:13:07.394169+0000 mgr.y (mgr.25390) 199 : cluster 0 pgmap v128: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 3 objects/s recovering 2023-07-10T17:13:08.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:08 smithi192 bash[50362]: audit 2023-07-10T17:13:07.402566+0000 mon.a (mon.0) 446 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:08.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:08 smithi192 bash[50362]: audit 2023-07-10T17:13:07.492414+0000 mon.a (mon.0) 447 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:13:08.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:08 smithi192 bash[50362]: audit 2023-07-10T17:13:07.496617+0000 mon.a (mon.0) 448 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:08.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:08 smithi192 bash[50362]: audit 2023-07-10T17:13:07.499029+0000 mon.a (mon.0) 449 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:08.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:08 smithi192 bash[50362]: audit 2023-07-10T17:13:07.500774+0000 mon.a (mon.0) 450 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:08.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:08 smithi192 bash[50362]: audit 2023-07-10T17:13:07.503197+0000 mon.a (mon.0) 451 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-07-10T17:13:08.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:08 smithi192 bash[50362]: audit 2023-07-10T17:13:07.503459+0000 mgr.y (mgr.25390) 200 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-07-10T17:13:08.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:08 smithi192 bash[50362]: cephadm 2023-07-10T17:13:07.505045+0000 mgr.y (mgr.25390) 201 : cephadm 1 Upgrade: osd.3 is safe to restart 2023-07-10T17:13:08.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[63900]: audit 2023-07-10T17:13:07.374411+0000 mon.a (mon.0) 442 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:08.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[63900]: audit 2023-07-10T17:13:07.386009+0000 mon.a (mon.0) 443 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:08.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[63900]: audit 2023-07-10T17:13:07.388344+0000 mon.a (mon.0) 444 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:13:08.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[63900]: audit 2023-07-10T17:13:07.390098+0000 mon.a (mon.0) 445 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:13:08.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[63900]: cephadm 2023-07-10T17:13:07.391578+0000 mgr.y (mgr.25390) 197 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:13:08.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[63900]: cluster 2023-07-10T17:13:07.393808+0000 mgr.y (mgr.25390) 198 : cluster 0 pgmap v127: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 3 objects/s recovering 2023-07-10T17:13:08.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[63900]: cluster 2023-07-10T17:13:07.394169+0000 mgr.y (mgr.25390) 199 : cluster 0 pgmap v128: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 3 objects/s recovering 2023-07-10T17:13:08.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[63900]: audit 2023-07-10T17:13:07.402566+0000 mon.a (mon.0) 446 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:08.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[63900]: audit 2023-07-10T17:13:07.492414+0000 mon.a (mon.0) 447 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:13:08.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[63900]: audit 2023-07-10T17:13:07.496617+0000 mon.a (mon.0) 448 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:08.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[63900]: audit 2023-07-10T17:13:07.499029+0000 mon.a (mon.0) 449 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:08.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[63900]: audit 2023-07-10T17:13:07.500774+0000 mon.a (mon.0) 450 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:08.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[63900]: audit 2023-07-10T17:13:07.503197+0000 mon.a (mon.0) 451 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-07-10T17:13:08.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[63900]: audit 2023-07-10T17:13:07.503459+0000 mgr.y (mgr.25390) 200 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-07-10T17:13:08.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[63900]: cephadm 2023-07-10T17:13:07.505045+0000 mgr.y (mgr.25390) 201 : cephadm 1 Upgrade: osd.3 is safe to restart 2023-07-10T17:13:08.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[65980]: audit 2023-07-10T17:13:07.374411+0000 mon.a (mon.0) 442 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:08.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[65980]: audit 2023-07-10T17:13:07.386009+0000 mon.a (mon.0) 443 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:08.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[65980]: audit 2023-07-10T17:13:07.388344+0000 mon.a (mon.0) 444 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:13:08.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[65980]: audit 2023-07-10T17:13:07.390098+0000 mon.a (mon.0) 445 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:13:08.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[65980]: cephadm 2023-07-10T17:13:07.391578+0000 mgr.y (mgr.25390) 197 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:13:08.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[65980]: cluster 2023-07-10T17:13:07.393808+0000 mgr.y (mgr.25390) 198 : cluster 0 pgmap v127: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 3 objects/s recovering 2023-07-10T17:13:08.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[65980]: cluster 2023-07-10T17:13:07.394169+0000 mgr.y (mgr.25390) 199 : cluster 0 pgmap v128: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 3 objects/s recovering 2023-07-10T17:13:08.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[65980]: audit 2023-07-10T17:13:07.402566+0000 mon.a (mon.0) 446 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:08.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[65980]: audit 2023-07-10T17:13:07.492414+0000 mon.a (mon.0) 447 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:13:08.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[65980]: audit 2023-07-10T17:13:07.496617+0000 mon.a (mon.0) 448 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:08.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[65980]: audit 2023-07-10T17:13:07.499029+0000 mon.a (mon.0) 449 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:08.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[65980]: audit 2023-07-10T17:13:07.500774+0000 mon.a (mon.0) 450 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:08.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[65980]: audit 2023-07-10T17:13:07.503197+0000 mon.a (mon.0) 451 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-07-10T17:13:08.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[65980]: audit 2023-07-10T17:13:07.503459+0000 mgr.y (mgr.25390) 200 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-07-10T17:13:08.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:08 smithi116 bash[65980]: cephadm 2023-07-10T17:13:07.505045+0000 mgr.y (mgr.25390) 201 : cephadm 1 Upgrade: osd.3 is safe to restart 2023-07-10T17:13:11.306 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:10 smithi116 bash[65980]: cluster 2023-07-10T17:13:09.394876+0000 mgr.y (mgr.25390) 202 : cluster 0 pgmap v129: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 3 objects/s recovering 2023-07-10T17:13:11.306 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:10 smithi116 bash[65980]: cephadm 2023-07-10T17:13:09.975936+0000 mgr.y (mgr.25390) 203 : cephadm 1 Upgrade: Updating osd.3 2023-07-10T17:13:11.306 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:10 smithi116 bash[65980]: audit 2023-07-10T17:13:09.985424+0000 mon.a (mon.0) 452 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:11.306 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:10 smithi116 bash[65980]: audit 2023-07-10T17:13:09.987423+0000 mon.a (mon.0) 453 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-10T17:13:11.307 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:10 smithi116 bash[65980]: audit 2023-07-10T17:13:09.988803+0000 mon.a (mon.0) 454 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:13:11.307 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:10 smithi116 bash[65980]: cephadm 2023-07-10T17:13:09.994381+0000 mgr.y (mgr.25390) 204 : cephadm 1 Deploying daemon osd.3 on smithi116 2023-07-10T17:13:11.307 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:10 smithi116 bash[63900]: cluster 2023-07-10T17:13:09.394876+0000 mgr.y (mgr.25390) 202 : cluster 0 pgmap v129: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 3 objects/s recovering 2023-07-10T17:13:11.307 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:10 smithi116 bash[63900]: cephadm 2023-07-10T17:13:09.975936+0000 mgr.y (mgr.25390) 203 : cephadm 1 Upgrade: Updating osd.3 2023-07-10T17:13:11.307 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:10 smithi116 bash[63900]: audit 2023-07-10T17:13:09.985424+0000 mon.a (mon.0) 452 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:11.307 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:10 smithi116 bash[63900]: audit 2023-07-10T17:13:09.987423+0000 mon.a (mon.0) 453 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-10T17:13:11.308 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:10 smithi116 bash[63900]: audit 2023-07-10T17:13:09.988803+0000 mon.a (mon.0) 454 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:13:11.308 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:10 smithi116 bash[63900]: cephadm 2023-07-10T17:13:09.994381+0000 mgr.y (mgr.25390) 204 : cephadm 1 Deploying daemon osd.3 on smithi116 2023-07-10T17:13:11.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:10 smithi192 bash[50362]: cluster 2023-07-10T17:13:09.394876+0000 mgr.y (mgr.25390) 202 : cluster 0 pgmap v129: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 3 objects/s recovering 2023-07-10T17:13:11.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:10 smithi192 bash[50362]: cephadm 2023-07-10T17:13:09.975936+0000 mgr.y (mgr.25390) 203 : cephadm 1 Upgrade: Updating osd.3 2023-07-10T17:13:11.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:10 smithi192 bash[50362]: audit 2023-07-10T17:13:09.985424+0000 mon.a (mon.0) 452 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:11.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:10 smithi192 bash[50362]: audit 2023-07-10T17:13:09.987423+0000 mon.a (mon.0) 453 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-10T17:13:11.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:10 smithi192 bash[50362]: audit 2023-07-10T17:13:09.988803+0000 mon.a (mon.0) 454 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:13:11.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:10 smithi192 bash[50362]: cephadm 2023-07-10T17:13:09.994381+0000 mgr.y (mgr.25390) 204 : cephadm 1 Deploying daemon osd.3 on smithi116 2023-07-10T17:13:11.391 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:13:12.130 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:13:12.131 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (6m) 15s ago 12m 10.8M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:13:12.131 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (5m) 2m ago 12m 25.6M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:13:12.131 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 error 15s ago 10m - - 2023-07-10T17:13:12.131 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (4m) 2m ago 16m 428M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:13:12.131 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (4m) 15s ago 18m 474M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:13:12.131 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (3m) 15s ago 18m 59.0M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:13:12.132 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (2m) 2m ago 17m 40.2M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 74a082a2e795 2023-07-10T17:13:12.132 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (3m) 15s ago 17m 51.1M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:13:12.132 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (6m) 15s ago 12m 9207k - 0.18.1 e5a616e4b9cf 3bd058d91353 2023-07-10T17:13:12.132 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (6m) 2m ago 12m 3607k - 0.18.1 e5a616e4b9cf 63551615c679 2023-07-10T17:13:12.132 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (80s) 15s ago 16m 51.4M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 0562f587fbca 2023-07-10T17:13:12.132 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (51s) 15s ago 16m 49.4M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 5b2a12a0bfc9 2023-07-10T17:13:12.132 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (20s) 15s ago 15m - 4096M 18.1.2-30-g8507faa7 17e8f0d79247 88096bc0faa9 2023-07-10T17:13:12.133 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (15m) 15s ago 15m 60.2M 4096M 16.2.0 24ecd6d5f14c 60949d479c11 2023-07-10T17:13:12.133 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (14m) 2m ago 14m 55.1M 4096M 16.2.0 24ecd6d5f14c 75fc0b36aab8 2023-07-10T17:13:12.133 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (14m) 2m ago 14m 52.8M 4096M 16.2.0 24ecd6d5f14c d673065333dd 2023-07-10T17:13:12.133 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (14m) 2m ago 14m 53.7M 4096M 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:13:12.133 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (13m) 2m ago 13m 56.7M 4096M 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:13:12.133 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 error 2m ago 12m - - 2023-07-10T17:13:12.133 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (10m) 15s ago 10m 55.7M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:13:12.134 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (10m) 2m ago 10m 54.8M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:13:12.134 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (10m) 15s ago 10m 55.3M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:13:12.134 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (10m) 2m ago 10m 56.6M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:13:12.447 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:12 smithi116 bash[63900]: audit 2023-07-10T17:13:11.371592+0000 mgr.y (mgr.25390) 205 : audit 0 from='client.44395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:12.447 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:12 smithi116 bash[63900]: cluster 2023-07-10T17:13:11.395481+0000 mgr.y (mgr.25390) 206 : cluster 0 pgmap v130: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 2 objects/s recovering 2023-07-10T17:13:12.447 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:12 smithi116 bash[63900]: audit 2023-07-10T17:13:11.444108+0000 mon.a (mon.0) 455 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:12.566 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:13:12.566 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:13:12.567 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:13:12.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:13:12.567 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:13:12.567 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:13:12.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:13:12.567 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:13:12.567 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 5, 2023-07-10T17:13:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:13:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:13:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:13:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:13:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:13:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:13:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 9, 2023-07-10T17:13:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 8 2023-07-10T17:13:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:13:12.569 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:13:12.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:12 smithi192 bash[50362]: audit 2023-07-10T17:13:11.371592+0000 mgr.y (mgr.25390) 205 : audit 0 from='client.44395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:12.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:12 smithi192 bash[50362]: cluster 2023-07-10T17:13:11.395481+0000 mgr.y (mgr.25390) 206 : cluster 0 pgmap v130: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 2 objects/s recovering 2023-07-10T17:13:12.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:12 smithi192 bash[50362]: audit 2023-07-10T17:13:11.444108+0000 mon.a (mon.0) 455 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:12.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:12 smithi192 bash[50362]: audit 2023-07-10T17:13:11.745629+0000 mgr.y (mgr.25390) 207 : audit 0 from='client.44401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:12.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:12 smithi192 bash[50362]: audit 2023-07-10T17:13:12.106867+0000 mgr.y (mgr.25390) 208 : audit 0 from='client.34551 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:12.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:12 smithi116 bash[63900]: audit 2023-07-10T17:13:11.745629+0000 mgr.y (mgr.25390) 207 : audit 0 from='client.44401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:12.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:12 smithi116 bash[63900]: audit 2023-07-10T17:13:12.106867+0000 mgr.y (mgr.25390) 208 : audit 0 from='client.34551 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:12.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:12 smithi116 bash[65980]: audit 2023-07-10T17:13:11.371592+0000 mgr.y (mgr.25390) 205 : audit 0 from='client.44395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:12.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:12 smithi116 bash[65980]: cluster 2023-07-10T17:13:11.395481+0000 mgr.y (mgr.25390) 206 : cluster 0 pgmap v130: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 2 objects/s recovering 2023-07-10T17:13:12.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:12 smithi116 bash[65980]: audit 2023-07-10T17:13:11.444108+0000 mon.a (mon.0) 455 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:12.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:12 smithi116 bash[65980]: audit 2023-07-10T17:13:11.745629+0000 mgr.y (mgr.25390) 207 : audit 0 from='client.44401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:12.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:12 smithi116 bash[65980]: audit 2023-07-10T17:13:12.106867+0000 mgr.y (mgr.25390) 208 : audit 0 from='client.34551 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:12.944 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:13:12.945 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:13:12.945 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:13:12.945 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:13:12.945 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:13:12.945 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2023-07-10T17:13:12.945 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2023-07-10T17:13:12.945 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:13:12.946 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "8/23 daemons upgraded", 2023-07-10T17:13:12.946 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading osd daemons", 2023-07-10T17:13:12.946 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:13:12.946 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:13:13.417 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:13:13.417 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:13:13.417 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:13:13.417 INFO:teuthology.orchestra.run.smithi116.stdout: daemon prometheus.a on smithi192 is in error state 2023-07-10T17:13:13.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:13 smithi116 bash[63900]: audit 2023-07-10T17:13:12.562194+0000 mon.a (mon.0) 456 : audit 0 from='client.34557 172.21.15.116:0/3097804229' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:13.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:13 smithi116 bash[63900]: audit 2023-07-10T17:13:12.939525+0000 mgr.y (mgr.25390) 209 : audit 0 from='client.34563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:13.572 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:13 smithi116 bash[63900]: audit 2023-07-10T17:13:13.412966+0000 mon.a (mon.0) 457 : audit 0 from='client.44425 172.21.15.116:0/4232471618' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:13:13.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:13 smithi192 bash[50362]: audit 2023-07-10T17:13:12.562194+0000 mon.a (mon.0) 456 : audit 0 from='client.34557 172.21.15.116:0/3097804229' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:13.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:13 smithi192 bash[50362]: audit 2023-07-10T17:13:12.939525+0000 mgr.y (mgr.25390) 209 : audit 0 from='client.34563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:13.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:13 smithi192 bash[50362]: audit 2023-07-10T17:13:13.412966+0000 mon.a (mon.0) 457 : audit 0 from='client.44425 172.21.15.116:0/4232471618' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:13:13.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:13 smithi116 bash[65980]: audit 2023-07-10T17:13:12.562194+0000 mon.a (mon.0) 456 : audit 0 from='client.34557 172.21.15.116:0/3097804229' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:13.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:13 smithi116 bash[65980]: audit 2023-07-10T17:13:12.939525+0000 mgr.y (mgr.25390) 209 : audit 0 from='client.34563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:13.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:13 smithi116 bash[65980]: audit 2023-07-10T17:13:13.412966+0000 mon.a (mon.0) 457 : audit 0 from='client.44425 172.21.15.116:0/4232471618' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:13:14.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:14 smithi116 bash[63900]: cluster 2023-07-10T17:13:13.397034+0000 mgr.y (mgr.25390) 210 : cluster 0 pgmap v131: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:13:14.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:14 smithi116 bash[65980]: cluster 2023-07-10T17:13:13.397034+0000 mgr.y (mgr.25390) 210 : cluster 0 pgmap v131: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:13:14.645 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:14 smithi116 systemd[1]: Stopping Ceph osd.3 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:13:14.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:14 smithi192 bash[50362]: cluster 2023-07-10T17:13:13.397034+0000 mgr.y (mgr.25390) 210 : cluster 0 pgmap v131: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:13:15.144 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:14 smithi116 bash[39200]: debug 2023-07-10T17:13:14.734+0000 7fbe1b0f8700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:13:15.144 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:14 smithi116 bash[39200]: debug 2023-07-10T17:13:14.734+0000 7fbe1b0f8700 -1 osd.3 92 *** Got signal Terminated *** 2023-07-10T17:13:15.144 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:14 smithi116 bash[39200]: debug 2023-07-10T17:13:14.734+0000 7fbe1b0f8700 -1 osd.3 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-10T17:13:15.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.012779+0000 mon.a (mon.0) 458 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-10T17:13:15.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.012909+0000 mon.a (mon.0) 459 : cluster 1 osd.3 failed (root=default,host=smithi116) (connection refused reported by osd.1) 2023-07-10T17:13:15.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.013374+0000 mon.a (mon.0) 460 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-10T17:13:15.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.013669+0000 mon.a (mon.0) 461 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-10T17:13:15.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.013923+0000 mon.a (mon.0) 462 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-10T17:13:15.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.014153+0000 mon.a (mon.0) 463 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-10T17:13:15.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.014365+0000 mon.a (mon.0) 464 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-10T17:13:15.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.014633+0000 mon.a (mon.0) 465 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-10T17:13:15.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.014841+0000 mon.a (mon.0) 466 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-10T17:13:15.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.015039+0000 mon.a (mon.0) 467 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-10T17:13:15.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.015270+0000 mon.a (mon.0) 468 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-10T17:13:15.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.015517+0000 mon.a (mon.0) 469 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-10T17:13:15.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.015702+0000 mon.a (mon.0) 470 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-10T17:13:15.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.015891+0000 mon.a (mon.0) 471 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-10T17:13:15.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.016078+0000 mon.a (mon.0) 472 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-10T17:13:15.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.416679+0000 mon.a (mon.0) 473 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-10T17:13:15.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.417194+0000 mon.a (mon.0) 474 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-10T17:13:15.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.417674+0000 mon.a (mon.0) 475 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-10T17:13:15.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.418082+0000 mon.a (mon.0) 476 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-10T17:13:15.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.418375+0000 mon.a (mon.0) 477 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-10T17:13:15.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.418580+0000 mon.a (mon.0) 478 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-10T17:13:15.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.418874+0000 mon.a (mon.0) 479 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-10T17:13:15.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.419109+0000 mon.a (mon.0) 480 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-10T17:13:15.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.419473+0000 mon.a (mon.0) 481 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-10T17:13:15.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.419612+0000 mon.a (mon.0) 482 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-10T17:13:15.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.420102+0000 mon.a (mon.0) 483 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-10T17:13:15.892 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.420421+0000 mon.a (mon.0) 484 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-10T17:13:15.892 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.420713+0000 mon.a (mon.0) 485 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-10T17:13:15.893 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:15 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.421026+0000 mon.a (mon.0) 486 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-10T17:13:15.894 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[76934]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-osd.3 2023-07-10T17:13:15.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.012779+0000 mon.a (mon.0) 458 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-10T17:13:15.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.012909+0000 mon.a (mon.0) 459 : cluster 1 osd.3 failed (root=default,host=smithi116) (connection refused reported by osd.1) 2023-07-10T17:13:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.013374+0000 mon.a (mon.0) 460 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-10T17:13:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.013669+0000 mon.a (mon.0) 461 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-10T17:13:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.013923+0000 mon.a (mon.0) 462 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-10T17:13:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.014153+0000 mon.a (mon.0) 463 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-10T17:13:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.014365+0000 mon.a (mon.0) 464 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-10T17:13:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.014633+0000 mon.a (mon.0) 465 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-10T17:13:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.014841+0000 mon.a (mon.0) 466 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-10T17:13:15.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.015039+0000 mon.a (mon.0) 467 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-10T17:13:15.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.015270+0000 mon.a (mon.0) 468 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-10T17:13:15.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.015517+0000 mon.a (mon.0) 469 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-10T17:13:15.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.015702+0000 mon.a (mon.0) 470 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-10T17:13:15.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.015891+0000 mon.a (mon.0) 471 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-10T17:13:15.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.016078+0000 mon.a (mon.0) 472 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-10T17:13:15.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.416679+0000 mon.a (mon.0) 473 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-10T17:13:15.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.417194+0000 mon.a (mon.0) 474 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-10T17:13:15.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.417674+0000 mon.a (mon.0) 475 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-10T17:13:15.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.418082+0000 mon.a (mon.0) 476 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-10T17:13:15.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.418375+0000 mon.a (mon.0) 477 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-10T17:13:15.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.418580+0000 mon.a (mon.0) 478 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-10T17:13:15.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.418874+0000 mon.a (mon.0) 479 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-10T17:13:15.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.419109+0000 mon.a (mon.0) 480 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-10T17:13:15.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.419473+0000 mon.a (mon.0) 481 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-10T17:13:15.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.419612+0000 mon.a (mon.0) 482 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-10T17:13:15.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.420102+0000 mon.a (mon.0) 483 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-10T17:13:15.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.420421+0000 mon.a (mon.0) 484 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-10T17:13:15.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.420713+0000 mon.a (mon.0) 485 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-10T17:13:15.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.421026+0000 mon.a (mon.0) 486 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-10T17:13:15.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.012779+0000 mon.a (mon.0) 458 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-10T17:13:15.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.012909+0000 mon.a (mon.0) 459 : cluster 1 osd.3 failed (root=default,host=smithi116) (connection refused reported by osd.1) 2023-07-10T17:13:15.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.013374+0000 mon.a (mon.0) 460 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-10T17:13:15.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.013669+0000 mon.a (mon.0) 461 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-10T17:13:15.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.013923+0000 mon.a (mon.0) 462 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-10T17:13:15.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.014153+0000 mon.a (mon.0) 463 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-10T17:13:15.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.014365+0000 mon.a (mon.0) 464 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-10T17:13:15.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.014633+0000 mon.a (mon.0) 465 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-10T17:13:15.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.014841+0000 mon.a (mon.0) 466 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-10T17:13:15.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.015039+0000 mon.a (mon.0) 467 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-10T17:13:15.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.015270+0000 mon.a (mon.0) 468 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-10T17:13:15.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.015517+0000 mon.a (mon.0) 469 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-10T17:13:15.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.015702+0000 mon.a (mon.0) 470 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-10T17:13:15.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.015891+0000 mon.a (mon.0) 471 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-10T17:13:15.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.016078+0000 mon.a (mon.0) 472 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-10T17:13:15.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.416679+0000 mon.a (mon.0) 473 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-10T17:13:15.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.417194+0000 mon.a (mon.0) 474 : cluster 0 osd.3 reported immediately failed by osd.1 2023-07-10T17:13:15.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.417674+0000 mon.a (mon.0) 475 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-10T17:13:15.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.418082+0000 mon.a (mon.0) 476 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-10T17:13:15.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.418375+0000 mon.a (mon.0) 477 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-10T17:13:15.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.418580+0000 mon.a (mon.0) 478 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-10T17:13:15.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.418874+0000 mon.a (mon.0) 479 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-10T17:13:15.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.419109+0000 mon.a (mon.0) 480 : cluster 0 osd.3 reported immediately failed by osd.7 2023-07-10T17:13:15.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.419473+0000 mon.a (mon.0) 481 : cluster 0 osd.3 reported immediately failed by osd.5 2023-07-10T17:13:15.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.419612+0000 mon.a (mon.0) 482 : cluster 0 osd.3 reported immediately failed by osd.4 2023-07-10T17:13:15.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.420102+0000 mon.a (mon.0) 483 : cluster 0 osd.3 reported immediately failed by osd.2 2023-07-10T17:13:15.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.420421+0000 mon.a (mon.0) 484 : cluster 0 osd.3 reported immediately failed by osd.0 2023-07-10T17:13:15.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.420713+0000 mon.a (mon.0) 485 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-10T17:13:15.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:15 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.421026+0000 mon.a (mon.0) 486 : cluster 0 osd.3 reported immediately failed by osd.6 2023-07-10T17:13:16.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.397943+0000 mgr.y (mgr.25390) 211 : cluster 0 pgmap v132: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:13:16.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.569404+0000 mon.a (mon.0) 487 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:13:16.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.585950+0000 osd.7 (osd.7) 19 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.588907+0000 osd.7 (osd.7) 20 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.589286+0000 osd.4 (osd.4) 17 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.590940+0000 osd.5 (osd.5) 17 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.591254+0000 osd.6 (osd.6) 23 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.592837+0000 mon.a (mon.0) 488 : cluster 0 osdmap e93: 8 total, 7 up, 8 in 2023-07-10T17:13:16.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.594203+0000 osd.7 (osd.7) 21 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.594204+0000 osd.4 (osd.4) 18 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.594835+0000 osd.6 (osd.6) 24 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.596929+0000 osd.5 (osd.5) 18 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.597813+0000 osd.7 (osd.7) 22 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.598151+0000 osd.4 (osd.4) 19 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.599319+0000 osd.5 (osd.5) 19 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.603040+0000 osd.4 (osd.4) 20 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:16 smithi192 bash[50362]: cluster 2023-07-10T17:13:15.605890+0000 osd.5 (osd.5) 20 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.397943+0000 mgr.y (mgr.25390) 211 : cluster 0 pgmap v132: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:13:16.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.569404+0000 mon.a (mon.0) 487 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:13:16.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.585950+0000 osd.7 (osd.7) 19 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.588907+0000 osd.7 (osd.7) 20 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.589286+0000 osd.4 (osd.4) 17 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.590940+0000 osd.5 (osd.5) 17 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.591254+0000 osd.6 (osd.6) 23 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.592837+0000 mon.a (mon.0) 488 : cluster 0 osdmap e93: 8 total, 7 up, 8 in 2023-07-10T17:13:16.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.594203+0000 osd.7 (osd.7) 21 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.594204+0000 osd.4 (osd.4) 18 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.594835+0000 osd.6 (osd.6) 24 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.596929+0000 osd.5 (osd.5) 18 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.597813+0000 osd.7 (osd.7) 22 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.598151+0000 osd.4 (osd.4) 19 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.599319+0000 osd.5 (osd.5) 19 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.603040+0000 osd.4 (osd.4) 20 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[65980]: cluster 2023-07-10T17:13:15.605890+0000 osd.5 (osd.5) 20 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.397943+0000 mgr.y (mgr.25390) 211 : cluster 0 pgmap v132: 161 pgs: 161 active+clean; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:13:16.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.569404+0000 mon.a (mon.0) 487 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:13:16.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.585950+0000 osd.7 (osd.7) 19 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.588907+0000 osd.7 (osd.7) 20 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.589286+0000 osd.4 (osd.4) 17 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.590940+0000 osd.5 (osd.5) 17 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.591254+0000 osd.6 (osd.6) 23 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.592837+0000 mon.a (mon.0) 488 : cluster 0 osdmap e93: 8 total, 7 up, 8 in 2023-07-10T17:13:16.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.594203+0000 osd.7 (osd.7) 21 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.594204+0000 osd.4 (osd.4) 18 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.594835+0000 osd.6 (osd.6) 24 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.596929+0000 osd.5 (osd.5) 18 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.597813+0000 osd.7 (osd.7) 22 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.598151+0000 osd.4 (osd.4) 19 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.599319+0000 osd.5 (osd.5) 19 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.603040+0000 osd.4 (osd.4) 20 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:16.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:16 smithi116 bash[63900]: cluster 2023-07-10T17:13:15.605890+0000 osd.5 (osd.5) 20 : cluster 3 failed to encode map e93 with expected crc 2023-07-10T17:13:17.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:17 smithi192 bash[50362]: cluster 2023-07-10T17:13:16.586854+0000 osd.4 (osd.4) 21 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:17 smithi192 bash[50362]: cluster 2023-07-10T17:13:16.587187+0000 osd.6 (osd.6) 25 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:17 smithi192 bash[50362]: cluster 2023-07-10T17:13:16.588654+0000 mon.a (mon.0) 489 : cluster 0 osdmap e94: 8 total, 7 up, 8 in 2023-07-10T17:13:17.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:17 smithi192 bash[50362]: cluster 2023-07-10T17:13:16.590251+0000 osd.4 (osd.4) 22 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:17 smithi192 bash[50362]: cluster 2023-07-10T17:13:16.590269+0000 osd.7 (osd.7) 23 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:17 smithi192 bash[50362]: cluster 2023-07-10T17:13:16.590307+0000 osd.5 (osd.5) 21 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:17 smithi192 bash[50362]: cluster 2023-07-10T17:13:16.590474+0000 osd.6 (osd.6) 26 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:17 smithi192 bash[50362]: cluster 2023-07-10T17:13:16.595482+0000 osd.7 (osd.7) 24 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:17 smithi192 bash[50362]: cluster 2023-07-10T17:13:16.595678+0000 osd.5 (osd.5) 22 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[63900]: cluster 2023-07-10T17:13:16.586854+0000 osd.4 (osd.4) 21 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[63900]: cluster 2023-07-10T17:13:16.587187+0000 osd.6 (osd.6) 25 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[63900]: cluster 2023-07-10T17:13:16.588654+0000 mon.a (mon.0) 489 : cluster 0 osdmap e94: 8 total, 7 up, 8 in 2023-07-10T17:13:17.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[63900]: cluster 2023-07-10T17:13:16.590251+0000 osd.4 (osd.4) 22 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[63900]: cluster 2023-07-10T17:13:16.590269+0000 osd.7 (osd.7) 23 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[63900]: cluster 2023-07-10T17:13:16.590307+0000 osd.5 (osd.5) 21 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[63900]: cluster 2023-07-10T17:13:16.590474+0000 osd.6 (osd.6) 26 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[63900]: cluster 2023-07-10T17:13:16.595482+0000 osd.7 (osd.7) 24 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[63900]: cluster 2023-07-10T17:13:16.595678+0000 osd.5 (osd.5) 22 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[65980]: cluster 2023-07-10T17:13:16.586854+0000 osd.4 (osd.4) 21 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[65980]: cluster 2023-07-10T17:13:16.587187+0000 osd.6 (osd.6) 25 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[65980]: cluster 2023-07-10T17:13:16.588654+0000 mon.a (mon.0) 489 : cluster 0 osdmap e94: 8 total, 7 up, 8 in 2023-07-10T17:13:17.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[65980]: cluster 2023-07-10T17:13:16.590251+0000 osd.4 (osd.4) 22 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[65980]: cluster 2023-07-10T17:13:16.590269+0000 osd.7 (osd.7) 23 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[65980]: cluster 2023-07-10T17:13:16.590307+0000 osd.5 (osd.5) 21 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[65980]: cluster 2023-07-10T17:13:16.590474+0000 osd.6 (osd.6) 26 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[65980]: cluster 2023-07-10T17:13:16.595482+0000 osd.7 (osd.7) 24 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:17 smithi116 bash[65980]: cluster 2023-07-10T17:13:16.595678+0000 osd.5 (osd.5) 22 : cluster 3 failed to encode map e94 with expected crc 2023-07-10T17:13:17.897 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:17 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.3.service: Succeeded. 2023-07-10T17:13:17.897 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:17 smithi116 systemd[1]: Stopped Ceph osd.3 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:13:18.396 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:17 smithi116 systemd[1]: Started Ceph osd.3 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:13:18.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:18 smithi192 bash[50362]: cluster 2023-07-10T17:13:17.399727+0000 mgr.y (mgr.25390) 212 : cluster 0 pgmap v135: 161 pgs: 1 activating+undersized+degraded, 95 active+clean, 23 active+undersized, 20 peering, 12 active+undersized+degraded, 10 activating+undersized; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 105/999 objects degraded (10.511%) 2023-07-10T17:13:18.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:18 smithi192 bash[50362]: cluster 2023-07-10T17:13:17.584809+0000 mon.a (mon.0) 490 : cluster 3 Health check failed: Reduced data availability: 3 pgs inactive (PG_AVAILABILITY) 2023-07-10T17:13:18.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:18 smithi192 bash[50362]: cluster 2023-07-10T17:13:17.584890+0000 mon.a (mon.0) 491 : cluster 3 Health check failed: Degraded data redundancy: 105/999 objects degraded (10.511%), 13 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:18.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:18 smithi192 bash[50362]: audit 2023-07-10T17:13:18.023270+0000 mon.a (mon.0) 492 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:18.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:18 smithi192 bash[50362]: audit 2023-07-10T17:13:18.030343+0000 mon.a (mon.0) 493 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:18.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:18 smithi116 bash[63900]: cluster 2023-07-10T17:13:17.399727+0000 mgr.y (mgr.25390) 212 : cluster 0 pgmap v135: 161 pgs: 1 activating+undersized+degraded, 95 active+clean, 23 active+undersized, 20 peering, 12 active+undersized+degraded, 10 activating+undersized; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 105/999 objects degraded (10.511%) 2023-07-10T17:13:18.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:18 smithi116 bash[63900]: cluster 2023-07-10T17:13:17.584809+0000 mon.a (mon.0) 490 : cluster 3 Health check failed: Reduced data availability: 3 pgs inactive (PG_AVAILABILITY) 2023-07-10T17:13:18.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:18 smithi116 bash[63900]: cluster 2023-07-10T17:13:17.584890+0000 mon.a (mon.0) 491 : cluster 3 Health check failed: Degraded data redundancy: 105/999 objects degraded (10.511%), 13 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:18.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:18 smithi116 bash[63900]: audit 2023-07-10T17:13:18.023270+0000 mon.a (mon.0) 492 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:18.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:18 smithi116 bash[63900]: audit 2023-07-10T17:13:18.030343+0000 mon.a (mon.0) 493 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:18.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:18 smithi116 bash[65980]: cluster 2023-07-10T17:13:17.399727+0000 mgr.y (mgr.25390) 212 : cluster 0 pgmap v135: 161 pgs: 1 activating+undersized+degraded, 95 active+clean, 23 active+undersized, 20 peering, 12 active+undersized+degraded, 10 activating+undersized; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 105/999 objects degraded (10.511%) 2023-07-10T17:13:18.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:18 smithi116 bash[65980]: cluster 2023-07-10T17:13:17.584809+0000 mon.a (mon.0) 490 : cluster 3 Health check failed: Reduced data availability: 3 pgs inactive (PG_AVAILABILITY) 2023-07-10T17:13:18.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:18 smithi116 bash[65980]: cluster 2023-07-10T17:13:17.584890+0000 mon.a (mon.0) 491 : cluster 3 Health check failed: Degraded data redundancy: 105/999 objects degraded (10.511%), 13 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:18.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:18 smithi116 bash[65980]: audit 2023-07-10T17:13:18.023270+0000 mon.a (mon.0) 492 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:18.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:18 smithi116 bash[65980]: audit 2023-07-10T17:13:18.030343+0000 mon.a (mon.0) 493 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:20.144 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:19 smithi116 bash[77199]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-07-10T17:13:20.144 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:19 smithi116 bash[77199]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2023-07-10T17:13:20.144 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:19 smithi116 bash[77199]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2023-07-10T17:13:20.144 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:19 smithi116 bash[77199]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2023-07-10T17:13:20.145 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:19 smithi116 bash[77199]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2023-07-10T17:13:20.145 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:19 smithi116 bash[77199]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-07-10T17:13:20.145 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:19 smithi116 bash[77199]: --> ceph-volume raw activate successful for osd ID: 3 2023-07-10T17:13:20.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:20 smithi192 bash[50362]: cluster 2023-07-10T17:13:19.400520+0000 mgr.y (mgr.25390) 213 : cluster 0 pgmap v136: 161 pgs: 1 activating+undersized+degraded, 95 active+clean, 23 active+undersized, 20 peering, 12 active+undersized+degraded, 10 activating+undersized; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 105/999 objects degraded (10.511%) 2023-07-10T17:13:20.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:20 smithi116 bash[63900]: cluster 2023-07-10T17:13:19.400520+0000 mgr.y (mgr.25390) 213 : cluster 0 pgmap v136: 161 pgs: 1 activating+undersized+degraded, 95 active+clean, 23 active+undersized, 20 peering, 12 active+undersized+degraded, 10 activating+undersized; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 105/999 objects degraded (10.511%) 2023-07-10T17:13:20.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:20 smithi116 bash[65980]: cluster 2023-07-10T17:13:19.400520+0000 mgr.y (mgr.25390) 213 : cluster 0 pgmap v136: 161 pgs: 1 activating+undersized+degraded, 95 active+clean, 23 active+undersized, 20 peering, 12 active+undersized+degraded, 10 activating+undersized; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 105/999 objects degraded (10.511%) 2023-07-10T17:13:21.644 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:21 smithi116 bash[77480]: debug 2023-07-10T17:13:21.469+0000 7f2aa4d57700 1 -- 172.21.15.116:0/1670356447 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56547b35c4e0 con 0x56547b3b6800 2023-07-10T17:13:22.394 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:22 smithi116 bash[77480]: debug 2023-07-10T17:13:22.037+0000 7f2aadedc640 -1 Falling back to public interface 2023-07-10T17:13:22.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:22 smithi192 bash[50362]: cluster 2023-07-10T17:13:21.401829+0000 mgr.y (mgr.25390) 214 : cluster 0 pgmap v137: 161 pgs: 1 activating+undersized+degraded, 95 active+clean, 37 active+undersized, 18 active+undersized+degraded, 10 activating+undersized; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 133/999 objects degraded (13.313%) 2023-07-10T17:13:22.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:22 smithi116 bash[63900]: cluster 2023-07-10T17:13:21.401829+0000 mgr.y (mgr.25390) 214 : cluster 0 pgmap v137: 161 pgs: 1 activating+undersized+degraded, 95 active+clean, 37 active+undersized, 18 active+undersized+degraded, 10 activating+undersized; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 133/999 objects degraded (13.313%) 2023-07-10T17:13:22.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:22 smithi116 bash[65980]: cluster 2023-07-10T17:13:21.401829+0000 mgr.y (mgr.25390) 214 : cluster 0 pgmap v137: 161 pgs: 1 activating+undersized+degraded, 95 active+clean, 37 active+undersized, 18 active+undersized+degraded, 10 activating+undersized; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 133/999 objects degraded (13.313%) 2023-07-10T17:13:24.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:23 smithi192 bash[50362]: cluster 2023-07-10T17:13:23.605212+0000 mon.a (mon.0) 494 : cluster 3 Health check update: Degraded data redundancy: 133/999 objects degraded (13.313%), 22 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:24.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:23 smithi192 bash[50362]: cluster 2023-07-10T17:13:23.605286+0000 mon.a (mon.0) 495 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive) 2023-07-10T17:13:24.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:23 smithi116 bash[63900]: cluster 2023-07-10T17:13:23.605212+0000 mon.a (mon.0) 494 : cluster 3 Health check update: Degraded data redundancy: 133/999 objects degraded (13.313%), 22 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:24.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:23 smithi116 bash[63900]: cluster 2023-07-10T17:13:23.605286+0000 mon.a (mon.0) 495 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive) 2023-07-10T17:13:24.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:23 smithi116 bash[65980]: cluster 2023-07-10T17:13:23.605212+0000 mon.a (mon.0) 494 : cluster 3 Health check update: Degraded data redundancy: 133/999 objects degraded (13.313%), 22 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:24.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:23 smithi116 bash[65980]: cluster 2023-07-10T17:13:23.605286+0000 mon.a (mon.0) 495 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive) 2023-07-10T17:13:25.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:24 smithi192 bash[50362]: cluster 2023-07-10T17:13:23.403664+0000 mgr.y (mgr.25390) 215 : cluster 0 pgmap v138: 161 pgs: 95 active+clean, 44 active+undersized, 22 active+undersized+degraded; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 133/999 objects degraded (13.313%) 2023-07-10T17:13:25.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:24 smithi116 bash[63900]: cluster 2023-07-10T17:13:23.403664+0000 mgr.y (mgr.25390) 215 : cluster 0 pgmap v138: 161 pgs: 95 active+clean, 44 active+undersized, 22 active+undersized+degraded; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 133/999 objects degraded (13.313%) 2023-07-10T17:13:25.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:24 smithi116 bash[65980]: cluster 2023-07-10T17:13:23.403664+0000 mgr.y (mgr.25390) 215 : cluster 0 pgmap v138: 161 pgs: 95 active+clean, 44 active+undersized, 22 active+undersized+degraded; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 133/999 objects degraded (13.313%) 2023-07-10T17:13:25.644 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:25 smithi116 bash[77480]: debug 2023-07-10T17:13:25.193+0000 7f2aadedc640 -1 osd.3 92 log_to_monitors true 2023-07-10T17:13:26.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:25 smithi192 bash[50362]: audit 2023-07-10T17:13:25.200092+0000 mon.a (mon.0) 496 : audit 1 from='osd.3 [v2:172.21.15.116:6826/2156776675,v1:172.21.15.116:6827/2156776675]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-10T17:13:26.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:25 smithi192 bash[50362]: audit 2023-07-10T17:13:25.580137+0000 mon.a (mon.0) 497 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:26.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:25 smithi192 bash[50362]: audit 2023-07-10T17:13:25.590840+0000 mon.a (mon.0) 498 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:26.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:25 smithi116 bash[63900]: audit 2023-07-10T17:13:25.200092+0000 mon.a (mon.0) 496 : audit 1 from='osd.3 [v2:172.21.15.116:6826/2156776675,v1:172.21.15.116:6827/2156776675]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-10T17:13:26.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:25 smithi116 bash[63900]: audit 2023-07-10T17:13:25.580137+0000 mon.a (mon.0) 497 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:26.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:25 smithi116 bash[63900]: audit 2023-07-10T17:13:25.590840+0000 mon.a (mon.0) 498 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:26.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:25 smithi116 bash[65980]: audit 2023-07-10T17:13:25.200092+0000 mon.a (mon.0) 496 : audit 1 from='osd.3 [v2:172.21.15.116:6826/2156776675,v1:172.21.15.116:6827/2156776675]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-10T17:13:26.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:25 smithi116 bash[65980]: audit 2023-07-10T17:13:25.580137+0000 mon.a (mon.0) 497 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:26.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:25 smithi116 bash[65980]: audit 2023-07-10T17:13:25.590840+0000 mon.a (mon.0) 498 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:26.145 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:13:25 smithi116 bash[77480]: debug 2023-07-10T17:13:25.741+0000 7f2a9c343700 -1 osd.3 92 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-10T17:13:27.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:26 smithi192 bash[50362]: cluster 2023-07-10T17:13:25.404897+0000 mgr.y (mgr.25390) 216 : cluster 0 pgmap v139: 161 pgs: 95 active+clean, 44 active+undersized, 22 active+undersized+degraded; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 133/999 objects degraded (13.313%) 2023-07-10T17:13:27.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:26 smithi192 bash[50362]: cluster 2023-07-10T17:13:25.675919+0000 osd.4 (osd.4) 23 : cluster 3 failed to encode map e95 with expected crc 2023-07-10T17:13:27.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:26 smithi192 bash[50362]: audit 2023-07-10T17:13:25.677843+0000 mon.a (mon.0) 499 : audit 1 from='osd.3 [v2:172.21.15.116:6826/2156776675,v1:172.21.15.116:6827/2156776675]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-10T17:13:27.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:26 smithi192 bash[50362]: cluster 2023-07-10T17:13:25.677950+0000 mon.a (mon.0) 500 : cluster 0 osdmap e95: 8 total, 7 up, 8 in 2023-07-10T17:13:27.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:26 smithi192 bash[50362]: audit 2023-07-10T17:13:25.678742+0000 mon.a (mon.0) 501 : audit 1 from='osd.3 [v2:172.21.15.116:6826/2156776675,v1:172.21.15.116:6827/2156776675]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:13:27.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:26 smithi192 bash[50362]: cluster 2023-07-10T17:13:26.215528+0000 osd.5 (osd.5) 23 : cluster 3 failed to encode map e95 with expected crc 2023-07-10T17:13:27.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:26 smithi192 bash[50362]: audit 2023-07-10T17:13:26.392070+0000 mon.a (mon.0) 502 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:27.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:26 smithi192 bash[50362]: audit 2023-07-10T17:13:26.404192+0000 mon.a (mon.0) 503 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:27.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:26 smithi116 bash[63900]: cluster 2023-07-10T17:13:25.404897+0000 mgr.y (mgr.25390) 216 : cluster 0 pgmap v139: 161 pgs: 95 active+clean, 44 active+undersized, 22 active+undersized+degraded; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 133/999 objects degraded (13.313%) 2023-07-10T17:13:27.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:26 smithi116 bash[63900]: cluster 2023-07-10T17:13:25.675919+0000 osd.4 (osd.4) 23 : cluster 3 failed to encode map e95 with expected crc 2023-07-10T17:13:27.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:26 smithi116 bash[63900]: audit 2023-07-10T17:13:25.677843+0000 mon.a (mon.0) 499 : audit 1 from='osd.3 [v2:172.21.15.116:6826/2156776675,v1:172.21.15.116:6827/2156776675]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-10T17:13:27.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:26 smithi116 bash[63900]: cluster 2023-07-10T17:13:25.677950+0000 mon.a (mon.0) 500 : cluster 0 osdmap e95: 8 total, 7 up, 8 in 2023-07-10T17:13:27.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:26 smithi116 bash[63900]: audit 2023-07-10T17:13:25.678742+0000 mon.a (mon.0) 501 : audit 1 from='osd.3 [v2:172.21.15.116:6826/2156776675,v1:172.21.15.116:6827/2156776675]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:13:27.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:26 smithi116 bash[63900]: cluster 2023-07-10T17:13:26.215528+0000 osd.5 (osd.5) 23 : cluster 3 failed to encode map e95 with expected crc 2023-07-10T17:13:27.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:26 smithi116 bash[63900]: audit 2023-07-10T17:13:26.392070+0000 mon.a (mon.0) 502 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:27.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:26 smithi116 bash[63900]: audit 2023-07-10T17:13:26.404192+0000 mon.a (mon.0) 503 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:27.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:26 smithi116 bash[65980]: cluster 2023-07-10T17:13:25.404897+0000 mgr.y (mgr.25390) 216 : cluster 0 pgmap v139: 161 pgs: 95 active+clean, 44 active+undersized, 22 active+undersized+degraded; 604 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 133/999 objects degraded (13.313%) 2023-07-10T17:13:27.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:26 smithi116 bash[65980]: cluster 2023-07-10T17:13:25.675919+0000 osd.4 (osd.4) 23 : cluster 3 failed to encode map e95 with expected crc 2023-07-10T17:13:27.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:26 smithi116 bash[65980]: audit 2023-07-10T17:13:25.677843+0000 mon.a (mon.0) 499 : audit 1 from='osd.3 [v2:172.21.15.116:6826/2156776675,v1:172.21.15.116:6827/2156776675]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-10T17:13:27.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:26 smithi116 bash[65980]: cluster 2023-07-10T17:13:25.677950+0000 mon.a (mon.0) 500 : cluster 0 osdmap e95: 8 total, 7 up, 8 in 2023-07-10T17:13:27.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:26 smithi116 bash[65980]: audit 2023-07-10T17:13:25.678742+0000 mon.a (mon.0) 501 : audit 1 from='osd.3 [v2:172.21.15.116:6826/2156776675,v1:172.21.15.116:6827/2156776675]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2023-07-10T17:13:27.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:26 smithi116 bash[65980]: cluster 2023-07-10T17:13:26.215528+0000 osd.5 (osd.5) 23 : cluster 3 failed to encode map e95 with expected crc 2023-07-10T17:13:27.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:26 smithi116 bash[65980]: audit 2023-07-10T17:13:26.392070+0000 mon.a (mon.0) 502 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:27.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:26 smithi116 bash[65980]: audit 2023-07-10T17:13:26.404192+0000 mon.a (mon.0) 503 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:28.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:27 smithi192 bash[50362]: cluster 2023-07-10T17:13:26.215566+0000 osd.6 (osd.6) 27 : cluster 3 failed to encode map e95 with expected crc 2023-07-10T17:13:28.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:27 smithi192 bash[50362]: cluster 2023-07-10T17:13:26.215607+0000 osd.7 (osd.7) 25 : cluster 3 failed to encode map e95 with expected crc 2023-07-10T17:13:28.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:27 smithi192 bash[50362]: cluster 2023-07-10T17:13:26.674109+0000 mon.a (mon.0) 504 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:13:28.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:27 smithi192 bash[50362]: cluster 2023-07-10T17:13:26.678761+0000 osd.4 (osd.4) 24 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:27 smithi192 bash[50362]: cluster 2023-07-10T17:13:26.682221+0000 osd.4 (osd.4) 25 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:27 smithi192 bash[50362]: cluster 2023-07-10T17:13:26.682945+0000 mon.a (mon.0) 505 : cluster 1 osd.3 [v2:172.21.15.116:6826/2156776675,v1:172.21.15.116:6827/2156776675] boot 2023-07-10T17:13:28.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:27 smithi192 bash[50362]: cluster 2023-07-10T17:13:26.683017+0000 mon.a (mon.0) 506 : cluster 0 osdmap e96: 8 total, 8 up, 8 in 2023-07-10T17:13:28.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:27 smithi192 bash[50362]: cluster 2023-07-10T17:13:26.684137+0000 osd.7 (osd.7) 26 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:27 smithi192 bash[50362]: audit 2023-07-10T17:13:26.684815+0000 mon.a (mon.0) 507 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T17:13:28.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:27 smithi192 bash[50362]: cluster 2023-07-10T17:13:26.685712+0000 osd.4 (osd.4) 26 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:27 smithi192 bash[50362]: cluster 2023-07-10T17:13:26.686946+0000 osd.5 (osd.5) 24 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:27 smithi192 bash[50362]: cluster 2023-07-10T17:13:26.688867+0000 osd.4 (osd.4) 27 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:27 smithi192 bash[50362]: cluster 2023-07-10T17:13:26.690240+0000 osd.5 (osd.5) 25 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:27 smithi192 bash[50362]: cluster 2023-07-10T17:13:26.695271+0000 osd.5 (osd.5) 26 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:27 smithi192 bash[50362]: cluster 2023-07-10T17:13:27.685273+0000 mon.a (mon.0) 508 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2023-07-10T17:13:28.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[63900]: cluster 2023-07-10T17:13:26.215566+0000 osd.6 (osd.6) 27 : cluster 3 failed to encode map e95 with expected crc 2023-07-10T17:13:28.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[63900]: cluster 2023-07-10T17:13:26.215607+0000 osd.7 (osd.7) 25 : cluster 3 failed to encode map e95 with expected crc 2023-07-10T17:13:28.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[63900]: cluster 2023-07-10T17:13:26.674109+0000 mon.a (mon.0) 504 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:13:28.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[63900]: cluster 2023-07-10T17:13:26.678761+0000 osd.4 (osd.4) 24 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[63900]: cluster 2023-07-10T17:13:26.682221+0000 osd.4 (osd.4) 25 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[63900]: cluster 2023-07-10T17:13:26.682945+0000 mon.a (mon.0) 505 : cluster 1 osd.3 [v2:172.21.15.116:6826/2156776675,v1:172.21.15.116:6827/2156776675] boot 2023-07-10T17:13:28.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[63900]: cluster 2023-07-10T17:13:26.683017+0000 mon.a (mon.0) 506 : cluster 0 osdmap e96: 8 total, 8 up, 8 in 2023-07-10T17:13:28.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[63900]: cluster 2023-07-10T17:13:26.684137+0000 osd.7 (osd.7) 26 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[63900]: audit 2023-07-10T17:13:26.684815+0000 mon.a (mon.0) 507 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T17:13:28.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[63900]: cluster 2023-07-10T17:13:26.685712+0000 osd.4 (osd.4) 26 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[63900]: cluster 2023-07-10T17:13:26.686946+0000 osd.5 (osd.5) 24 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[63900]: cluster 2023-07-10T17:13:26.688867+0000 osd.4 (osd.4) 27 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[63900]: cluster 2023-07-10T17:13:26.690240+0000 osd.5 (osd.5) 25 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[63900]: cluster 2023-07-10T17:13:26.695271+0000 osd.5 (osd.5) 26 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[63900]: cluster 2023-07-10T17:13:27.685273+0000 mon.a (mon.0) 508 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2023-07-10T17:13:28.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[65980]: cluster 2023-07-10T17:13:26.215566+0000 osd.6 (osd.6) 27 : cluster 3 failed to encode map e95 with expected crc 2023-07-10T17:13:28.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[65980]: cluster 2023-07-10T17:13:26.215607+0000 osd.7 (osd.7) 25 : cluster 3 failed to encode map e95 with expected crc 2023-07-10T17:13:28.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[65980]: cluster 2023-07-10T17:13:26.674109+0000 mon.a (mon.0) 504 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:13:28.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[65980]: cluster 2023-07-10T17:13:26.678761+0000 osd.4 (osd.4) 24 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[65980]: cluster 2023-07-10T17:13:26.682221+0000 osd.4 (osd.4) 25 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[65980]: cluster 2023-07-10T17:13:26.682945+0000 mon.a (mon.0) 505 : cluster 1 osd.3 [v2:172.21.15.116:6826/2156776675,v1:172.21.15.116:6827/2156776675] boot 2023-07-10T17:13:28.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[65980]: cluster 2023-07-10T17:13:26.683017+0000 mon.a (mon.0) 506 : cluster 0 osdmap e96: 8 total, 8 up, 8 in 2023-07-10T17:13:28.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[65980]: cluster 2023-07-10T17:13:26.684137+0000 osd.7 (osd.7) 26 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[65980]: audit 2023-07-10T17:13:26.684815+0000 mon.a (mon.0) 507 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-10T17:13:28.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[65980]: cluster 2023-07-10T17:13:26.685712+0000 osd.4 (osd.4) 26 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[65980]: cluster 2023-07-10T17:13:26.686946+0000 osd.5 (osd.5) 24 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[65980]: cluster 2023-07-10T17:13:26.688867+0000 osd.4 (osd.4) 27 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[65980]: cluster 2023-07-10T17:13:26.690240+0000 osd.5 (osd.5) 25 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[65980]: cluster 2023-07-10T17:13:26.695271+0000 osd.5 (osd.5) 26 : cluster 3 failed to encode map e96 with expected crc 2023-07-10T17:13:28.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:27 smithi116 bash[65980]: cluster 2023-07-10T17:13:27.685273+0000 mon.a (mon.0) 508 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2023-07-10T17:13:29.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:28 smithi192 bash[50362]: cluster 2023-07-10T17:13:27.406499+0000 mgr.y (mgr.25390) 217 : cluster 0 pgmap v142: 161 pgs: 18 peering, 95 active+clean, 32 active+undersized, 16 active+undersized+degraded; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 105/999 objects degraded (10.511%) 2023-07-10T17:13:29.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:28 smithi192 bash[50362]: cluster 2023-07-10T17:13:27.685851+0000 osd.7 (osd.7) 27 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:28 smithi192 bash[50362]: cluster 2023-07-10T17:13:27.687128+0000 osd.5 (osd.5) 27 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:28 smithi192 bash[50362]: cluster 2023-07-10T17:13:27.688677+0000 osd.6 (osd.6) 28 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:28 smithi192 bash[50362]: cluster 2023-07-10T17:13:27.688984+0000 osd.7 (osd.7) 28 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:28 smithi192 bash[50362]: cluster 2023-07-10T17:13:27.689064+0000 osd.4 (osd.4) 28 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:28 smithi192 bash[50362]: cluster 2023-07-10T17:13:27.690755+0000 osd.5 (osd.5) 28 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:28 smithi192 bash[50362]: cluster 2023-07-10T17:13:27.694117+0000 osd.7 (osd.7) 29 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:28 smithi192 bash[50362]: cluster 2023-07-10T17:13:27.695805+0000 osd.4 (osd.4) 29 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:28 smithi192 bash[50362]: cluster 2023-07-10T17:13:27.696134+0000 osd.5 (osd.5) 29 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:28 smithi192 bash[50362]: cluster 2023-07-10T17:13:27.699149+0000 osd.4 (osd.4) 30 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:28 smithi192 bash[50362]: cluster 2023-07-10T17:13:27.701195+0000 osd.4 (osd.4) 31 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:28 smithi192 bash[50362]: cluster 2023-07-10T17:13:27.703108+0000 osd.4 (osd.4) 32 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[63900]: cluster 2023-07-10T17:13:27.406499+0000 mgr.y (mgr.25390) 217 : cluster 0 pgmap v142: 161 pgs: 18 peering, 95 active+clean, 32 active+undersized, 16 active+undersized+degraded; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 105/999 objects degraded (10.511%) 2023-07-10T17:13:29.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[63900]: cluster 2023-07-10T17:13:27.685851+0000 osd.7 (osd.7) 27 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[63900]: cluster 2023-07-10T17:13:27.687128+0000 osd.5 (osd.5) 27 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[63900]: cluster 2023-07-10T17:13:27.688677+0000 osd.6 (osd.6) 28 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[63900]: cluster 2023-07-10T17:13:27.688984+0000 osd.7 (osd.7) 28 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[63900]: cluster 2023-07-10T17:13:27.689064+0000 osd.4 (osd.4) 28 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[63900]: cluster 2023-07-10T17:13:27.690755+0000 osd.5 (osd.5) 28 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[63900]: cluster 2023-07-10T17:13:27.694117+0000 osd.7 (osd.7) 29 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[63900]: cluster 2023-07-10T17:13:27.695805+0000 osd.4 (osd.4) 29 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[63900]: cluster 2023-07-10T17:13:27.696134+0000 osd.5 (osd.5) 29 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[63900]: cluster 2023-07-10T17:13:27.699149+0000 osd.4 (osd.4) 30 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[63900]: cluster 2023-07-10T17:13:27.701195+0000 osd.4 (osd.4) 31 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[63900]: cluster 2023-07-10T17:13:27.703108+0000 osd.4 (osd.4) 32 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[65980]: cluster 2023-07-10T17:13:27.406499+0000 mgr.y (mgr.25390) 217 : cluster 0 pgmap v142: 161 pgs: 18 peering, 95 active+clean, 32 active+undersized, 16 active+undersized+degraded; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 105/999 objects degraded (10.511%) 2023-07-10T17:13:29.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[65980]: cluster 2023-07-10T17:13:27.685851+0000 osd.7 (osd.7) 27 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[65980]: cluster 2023-07-10T17:13:27.687128+0000 osd.5 (osd.5) 27 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[65980]: cluster 2023-07-10T17:13:27.688677+0000 osd.6 (osd.6) 28 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[65980]: cluster 2023-07-10T17:13:27.688984+0000 osd.7 (osd.7) 28 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[65980]: cluster 2023-07-10T17:13:27.689064+0000 osd.4 (osd.4) 28 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[65980]: cluster 2023-07-10T17:13:27.690755+0000 osd.5 (osd.5) 28 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[65980]: cluster 2023-07-10T17:13:27.694117+0000 osd.7 (osd.7) 29 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[65980]: cluster 2023-07-10T17:13:27.695805+0000 osd.4 (osd.4) 29 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[65980]: cluster 2023-07-10T17:13:27.696134+0000 osd.5 (osd.5) 29 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[65980]: cluster 2023-07-10T17:13:27.699149+0000 osd.4 (osd.4) 30 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[65980]: cluster 2023-07-10T17:13:27.701195+0000 osd.4 (osd.4) 31 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:29.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:28 smithi116 bash[65980]: cluster 2023-07-10T17:13:27.703108+0000 osd.4 (osd.4) 32 : cluster 3 failed to encode map e97 with expected crc 2023-07-10T17:13:31.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:30 smithi192 bash[50362]: cluster 2023-07-10T17:13:29.407508+0000 mgr.y (mgr.25390) 218 : cluster 0 pgmap v144: 161 pgs: 18 peering, 95 active+clean, 32 active+undersized, 16 active+undersized+degraded; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 105/999 objects degraded (10.511%) 2023-07-10T17:13:31.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:30 smithi116 bash[63900]: cluster 2023-07-10T17:13:29.407508+0000 mgr.y (mgr.25390) 218 : cluster 0 pgmap v144: 161 pgs: 18 peering, 95 active+clean, 32 active+undersized, 16 active+undersized+degraded; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 105/999 objects degraded (10.511%) 2023-07-10T17:13:31.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:30 smithi116 bash[65980]: cluster 2023-07-10T17:13:29.407508+0000 mgr.y (mgr.25390) 218 : cluster 0 pgmap v144: 161 pgs: 18 peering, 95 active+clean, 32 active+undersized, 16 active+undersized+degraded; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 105/999 objects degraded (10.511%) 2023-07-10T17:13:32.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:31 smithi192 bash[50362]: cluster 2023-07-10T17:13:31.714311+0000 mon.a (mon.0) 509 : cluster 3 Health check update: Degraded data redundancy: 27/999 objects degraded (2.703%), 3 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:32.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:31 smithi116 bash[63900]: cluster 2023-07-10T17:13:31.714311+0000 mon.a (mon.0) 509 : cluster 3 Health check update: Degraded data redundancy: 27/999 objects degraded (2.703%), 3 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:32.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:31 smithi116 bash[65980]: cluster 2023-07-10T17:13:31.714311+0000 mon.a (mon.0) 509 : cluster 3 Health check update: Degraded data redundancy: 27/999 objects degraded (2.703%), 3 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:32.835 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:32 smithi116 bash[63900]: cluster 2023-07-10T17:13:31.408728+0000 mgr.y (mgr.25390) 219 : cluster 0 pgmap v145: 161 pgs: 18 peering, 135 active+clean, 5 active+undersized, 3 active+undersized+degraded; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 27/999 objects degraded (2.703%) 2023-07-10T17:13:32.836 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:32 smithi116 bash[65980]: cluster 2023-07-10T17:13:31.408728+0000 mgr.y (mgr.25390) 219 : cluster 0 pgmap v145: 161 pgs: 18 peering, 135 active+clean, 5 active+undersized, 3 active+undersized+degraded; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 27/999 objects degraded (2.703%) 2023-07-10T17:13:33.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:32 smithi192 bash[50362]: cluster 2023-07-10T17:13:31.408728+0000 mgr.y (mgr.25390) 219 : cluster 0 pgmap v145: 161 pgs: 18 peering, 135 active+clean, 5 active+undersized, 3 active+undersized+degraded; 604 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 27/999 objects degraded (2.703%) 2023-07-10T17:13:34.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:33 smithi192 bash[50362]: cluster 2023-07-10T17:13:33.769550+0000 mon.a (mon.0) 510 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/999 objects degraded (2.703%), 3 pgs degraded) 2023-07-10T17:13:34.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:33 smithi116 bash[65980]: cluster 2023-07-10T17:13:33.769550+0000 mon.a (mon.0) 510 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/999 objects degraded (2.703%), 3 pgs degraded) 2023-07-10T17:13:34.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:33 smithi116 bash[63900]: cluster 2023-07-10T17:13:33.769550+0000 mon.a (mon.0) 510 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/999 objects degraded (2.703%), 3 pgs degraded) 2023-07-10T17:13:35.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:34 smithi192 bash[50362]: cluster 2023-07-10T17:13:33.409955+0000 mgr.y (mgr.25390) 220 : cluster 0 pgmap v146: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 132 B/s rd, 0 op/s 2023-07-10T17:13:35.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:34 smithi116 bash[65980]: cluster 2023-07-10T17:13:33.409955+0000 mgr.y (mgr.25390) 220 : cluster 0 pgmap v146: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 132 B/s rd, 0 op/s 2023-07-10T17:13:35.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:34 smithi116 bash[63900]: cluster 2023-07-10T17:13:33.409955+0000 mgr.y (mgr.25390) 220 : cluster 0 pgmap v146: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 132 B/s rd, 0 op/s 2023-07-10T17:13:37.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:36 smithi192 bash[50362]: cluster 2023-07-10T17:13:35.410717+0000 mgr.y (mgr.25390) 221 : cluster 0 pgmap v147: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 117 B/s rd, 0 op/s 2023-07-10T17:13:37.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:36 smithi192 bash[50362]: audit 2023-07-10T17:13:36.505934+0000 mon.a (mon.0) 511 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:37.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:36 smithi192 bash[50362]: audit 2023-07-10T17:13:36.517660+0000 mon.a (mon.0) 512 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:37.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:36 smithi192 bash[50362]: audit 2023-07-10T17:13:36.519922+0000 mon.a (mon.0) 513 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:13:37.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:36 smithi192 bash[50362]: audit 2023-07-10T17:13:36.521690+0000 mon.a (mon.0) 514 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:13:37.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:36 smithi192 bash[50362]: audit 2023-07-10T17:13:36.532701+0000 mon.a (mon.0) 515 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:37.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:36 smithi192 bash[50362]: audit 2023-07-10T17:13:36.622128+0000 mon.a (mon.0) 516 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:13:37.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:36 smithi192 bash[50362]: audit 2023-07-10T17:13:36.626391+0000 mon.a (mon.0) 517 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:37.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:36 smithi192 bash[50362]: audit 2023-07-10T17:13:36.628953+0000 mon.a (mon.0) 518 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:37.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:36 smithi192 bash[50362]: audit 2023-07-10T17:13:36.631047+0000 mon.a (mon.0) 519 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:37.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:36 smithi192 bash[50362]: audit 2023-07-10T17:13:36.634729+0000 mon.a (mon.0) 520 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-07-10T17:13:37.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[63900]: cluster 2023-07-10T17:13:35.410717+0000 mgr.y (mgr.25390) 221 : cluster 0 pgmap v147: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 117 B/s rd, 0 op/s 2023-07-10T17:13:37.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[63900]: audit 2023-07-10T17:13:36.505934+0000 mon.a (mon.0) 511 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:37.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[63900]: audit 2023-07-10T17:13:36.517660+0000 mon.a (mon.0) 512 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:37.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[63900]: audit 2023-07-10T17:13:36.519922+0000 mon.a (mon.0) 513 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:13:37.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[63900]: audit 2023-07-10T17:13:36.521690+0000 mon.a (mon.0) 514 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:13:37.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[63900]: audit 2023-07-10T17:13:36.532701+0000 mon.a (mon.0) 515 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:37.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[63900]: audit 2023-07-10T17:13:36.622128+0000 mon.a (mon.0) 516 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:13:37.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[63900]: audit 2023-07-10T17:13:36.626391+0000 mon.a (mon.0) 517 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:37.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[63900]: audit 2023-07-10T17:13:36.628953+0000 mon.a (mon.0) 518 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:37.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[63900]: audit 2023-07-10T17:13:36.631047+0000 mon.a (mon.0) 519 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:37.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[63900]: audit 2023-07-10T17:13:36.634729+0000 mon.a (mon.0) 520 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-07-10T17:13:37.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[65980]: cluster 2023-07-10T17:13:35.410717+0000 mgr.y (mgr.25390) 221 : cluster 0 pgmap v147: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 117 B/s rd, 0 op/s 2023-07-10T17:13:37.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[65980]: audit 2023-07-10T17:13:36.505934+0000 mon.a (mon.0) 511 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:37.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[65980]: audit 2023-07-10T17:13:36.517660+0000 mon.a (mon.0) 512 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:37.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[65980]: audit 2023-07-10T17:13:36.519922+0000 mon.a (mon.0) 513 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:13:37.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[65980]: audit 2023-07-10T17:13:36.521690+0000 mon.a (mon.0) 514 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:13:37.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[65980]: audit 2023-07-10T17:13:36.532701+0000 mon.a (mon.0) 515 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:37.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[65980]: audit 2023-07-10T17:13:36.622128+0000 mon.a (mon.0) 516 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:13:37.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[65980]: audit 2023-07-10T17:13:36.626391+0000 mon.a (mon.0) 517 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:37.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[65980]: audit 2023-07-10T17:13:36.628953+0000 mon.a (mon.0) 518 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:37.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[65980]: audit 2023-07-10T17:13:36.631047+0000 mon.a (mon.0) 519 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:37.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:36 smithi116 bash[65980]: audit 2023-07-10T17:13:36.634729+0000 mon.a (mon.0) 520 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-07-10T17:13:38.102 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:37 smithi192 bash[50362]: cephadm 2023-07-10T17:13:36.523254+0000 mgr.y (mgr.25390) 222 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:13:38.103 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:37 smithi192 bash[50362]: cluster 2023-07-10T17:13:36.524776+0000 mgr.y (mgr.25390) 223 : cluster 0 pgmap v148: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2023-07-10T17:13:38.103 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:37 smithi192 bash[50362]: cluster 2023-07-10T17:13:36.525231+0000 mgr.y (mgr.25390) 224 : cluster 0 pgmap v149: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 115 B/s rd, 0 op/s 2023-07-10T17:13:38.103 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:37 smithi192 bash[50362]: audit 2023-07-10T17:13:36.635080+0000 mgr.y (mgr.25390) 225 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-07-10T17:13:38.103 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:37 smithi192 bash[50362]: cephadm 2023-07-10T17:13:36.637312+0000 mgr.y (mgr.25390) 226 : cephadm 1 Upgrade: osd.4 is safe to restart 2023-07-10T17:13:38.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:37 smithi116 bash[63900]: cephadm 2023-07-10T17:13:36.523254+0000 mgr.y (mgr.25390) 222 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:13:38.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:37 smithi116 bash[63900]: cluster 2023-07-10T17:13:36.524776+0000 mgr.y (mgr.25390) 223 : cluster 0 pgmap v148: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2023-07-10T17:13:38.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:37 smithi116 bash[63900]: cluster 2023-07-10T17:13:36.525231+0000 mgr.y (mgr.25390) 224 : cluster 0 pgmap v149: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 115 B/s rd, 0 op/s 2023-07-10T17:13:38.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:37 smithi116 bash[63900]: audit 2023-07-10T17:13:36.635080+0000 mgr.y (mgr.25390) 225 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-07-10T17:13:38.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:37 smithi116 bash[63900]: cephadm 2023-07-10T17:13:36.637312+0000 mgr.y (mgr.25390) 226 : cephadm 1 Upgrade: osd.4 is safe to restart 2023-07-10T17:13:38.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:37 smithi116 bash[65980]: cephadm 2023-07-10T17:13:36.523254+0000 mgr.y (mgr.25390) 222 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:13:38.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:37 smithi116 bash[65980]: cluster 2023-07-10T17:13:36.524776+0000 mgr.y (mgr.25390) 223 : cluster 0 pgmap v148: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2023-07-10T17:13:38.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:37 smithi116 bash[65980]: cluster 2023-07-10T17:13:36.525231+0000 mgr.y (mgr.25390) 224 : cluster 0 pgmap v149: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 115 B/s rd, 0 op/s 2023-07-10T17:13:38.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:37 smithi116 bash[65980]: audit 2023-07-10T17:13:36.635080+0000 mgr.y (mgr.25390) 225 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-07-10T17:13:38.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:37 smithi116 bash[65980]: cephadm 2023-07-10T17:13:36.637312+0000 mgr.y (mgr.25390) 226 : cephadm 1 Upgrade: osd.4 is safe to restart 2023-07-10T17:13:40.284 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:39 smithi192 bash[50362]: cluster 2023-07-10T17:13:38.526609+0000 mgr.y (mgr.25390) 227 : cluster 0 pgmap v150: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2023-07-10T17:13:40.284 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:39 smithi192 bash[50362]: cephadm 2023-07-10T17:13:38.957607+0000 mgr.y (mgr.25390) 228 : cephadm 1 Upgrade: Updating osd.4 2023-07-10T17:13:40.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:39 smithi192 bash[50362]: audit 2023-07-10T17:13:38.967751+0000 mon.a (mon.0) 521 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:40.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:39 smithi192 bash[50362]: audit 2023-07-10T17:13:38.970749+0000 mon.a (mon.0) 522 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-10T17:13:40.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:39 smithi192 bash[50362]: audit 2023-07-10T17:13:38.982765+0000 mon.a (mon.0) 523 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:13:40.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:39 smithi192 bash[50362]: cephadm 2023-07-10T17:13:38.987095+0000 mgr.y (mgr.25390) 229 : cephadm 1 Deploying daemon osd.4 on smithi192 2023-07-10T17:13:40.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:39 smithi116 bash[65980]: cluster 2023-07-10T17:13:38.526609+0000 mgr.y (mgr.25390) 227 : cluster 0 pgmap v150: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2023-07-10T17:13:40.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:39 smithi116 bash[65980]: cephadm 2023-07-10T17:13:38.957607+0000 mgr.y (mgr.25390) 228 : cephadm 1 Upgrade: Updating osd.4 2023-07-10T17:13:40.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:39 smithi116 bash[65980]: audit 2023-07-10T17:13:38.967751+0000 mon.a (mon.0) 521 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:40.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:39 smithi116 bash[65980]: audit 2023-07-10T17:13:38.970749+0000 mon.a (mon.0) 522 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-10T17:13:40.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:39 smithi116 bash[65980]: audit 2023-07-10T17:13:38.982765+0000 mon.a (mon.0) 523 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:13:40.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:39 smithi116 bash[65980]: cephadm 2023-07-10T17:13:38.987095+0000 mgr.y (mgr.25390) 229 : cephadm 1 Deploying daemon osd.4 on smithi192 2023-07-10T17:13:40.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:39 smithi116 bash[63900]: cluster 2023-07-10T17:13:38.526609+0000 mgr.y (mgr.25390) 227 : cluster 0 pgmap v150: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2023-07-10T17:13:40.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:39 smithi116 bash[63900]: cephadm 2023-07-10T17:13:38.957607+0000 mgr.y (mgr.25390) 228 : cephadm 1 Upgrade: Updating osd.4 2023-07-10T17:13:40.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:39 smithi116 bash[63900]: audit 2023-07-10T17:13:38.967751+0000 mon.a (mon.0) 521 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:40.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:39 smithi116 bash[63900]: audit 2023-07-10T17:13:38.970749+0000 mon.a (mon.0) 522 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-10T17:13:40.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:39 smithi116 bash[63900]: audit 2023-07-10T17:13:38.982765+0000 mon.a (mon.0) 523 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:13:40.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:39 smithi116 bash[63900]: cephadm 2023-07-10T17:13:38.987095+0000 mgr.y (mgr.25390) 229 : cephadm 1 Deploying daemon osd.4 on smithi192 2023-07-10T17:13:42.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:41 smithi192 bash[50362]: cluster 2023-07-10T17:13:40.527403+0000 mgr.y (mgr.25390) 230 : cluster 0 pgmap v151: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:13:42.335 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:41 smithi192 bash[50362]: audit 2023-07-10T17:13:41.472626+0000 mon.a (mon.0) 524 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:42.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:41 smithi116 bash[63900]: cluster 2023-07-10T17:13:40.527403+0000 mgr.y (mgr.25390) 230 : cluster 0 pgmap v151: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:13:42.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:41 smithi116 bash[63900]: audit 2023-07-10T17:13:41.472626+0000 mon.a (mon.0) 524 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:42.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:41 smithi116 bash[65980]: cluster 2023-07-10T17:13:40.527403+0000 mgr.y (mgr.25390) 230 : cluster 0 pgmap v151: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:13:42.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:41 smithi116 bash[65980]: audit 2023-07-10T17:13:41.472626+0000 mon.a (mon.0) 524 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:43.387 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:43 smithi192 systemd[1]: Stopping Ceph osd.4 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:13:43.696 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[26676]: debug 2023-07-10T17:13:43.426+0000 7fbb5f2ad700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:13:43.696 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[26676]: debug 2023-07-10T17:13:43.426+0000 7fbb5f2ad700 -1 osd.4 97 *** Got signal Terminated *** 2023-07-10T17:13:43.696 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[26676]: debug 2023-07-10T17:13:43.426+0000 7fbb5f2ad700 -1 osd.4 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-10T17:13:43.826 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:13:44.241 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:42.529043+0000 mgr.y (mgr.25390) 231 : cluster 0 pgmap v152: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:13:44.242 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.482241+0000 mon.a (mon.0) 525 : cluster 0 osd.4 reported immediately failed by osd.7 2023-07-10T17:13:44.242 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.482338+0000 mon.a (mon.0) 526 : cluster 1 osd.4 failed (root=default,host=smithi192) (connection refused reported by osd.7) 2023-07-10T17:13:44.242 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.482851+0000 mon.a (mon.0) 527 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.242 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.483195+0000 mon.a (mon.0) 528 : cluster 0 osd.4 reported immediately failed by osd.0 2023-07-10T17:13:44.242 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.483658+0000 mon.a (mon.0) 529 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.242 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.483939+0000 mon.a (mon.0) 530 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.242 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.484373+0000 mon.a (mon.0) 531 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.243 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.531619+0000 mon.a (mon.0) 532 : cluster 0 osd.4 reported immediately failed by osd.6 2023-07-10T17:13:44.243 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.536653+0000 mon.a (mon.0) 533 : cluster 0 osd.4 reported immediately failed by osd.5 2023-07-10T17:13:44.243 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.539144+0000 mon.a (mon.0) 534 : cluster 0 osd.4 reported immediately failed by osd.7 2023-07-10T17:13:44.243 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.684249+0000 mon.a (mon.0) 535 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.243 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.684567+0000 mon.a (mon.0) 536 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.243 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.684901+0000 mon.a (mon.0) 537 : cluster 0 osd.4 reported immediately failed by osd.0 2023-07-10T17:13:44.243 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.685186+0000 mon.a (mon.0) 538 : cluster 0 osd.4 reported immediately failed by osd.0 2023-07-10T17:13:44.243 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.685697+0000 mon.a (mon.0) 539 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.244 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.685976+0000 mon.a (mon.0) 540 : cluster 0 osd.4 reported immediately failed by osd.7 2023-07-10T17:13:44.244 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.686167+0000 mon.a (mon.0) 541 : cluster 0 osd.4 reported immediately failed by osd.5 2023-07-10T17:13:44.244 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.686412+0000 mon.a (mon.0) 542 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.244 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.686690+0000 mon.a (mon.0) 543 : cluster 0 osd.4 reported immediately failed by osd.7 2023-07-10T17:13:44.244 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.686883+0000 mon.a (mon.0) 544 : cluster 0 osd.4 reported immediately failed by osd.5 2023-07-10T17:13:44.244 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.687130+0000 mon.a (mon.0) 545 : cluster 0 osd.4 reported immediately failed by osd.2 2023-07-10T17:13:44.244 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.687494+0000 mon.a (mon.0) 546 : cluster 0 osd.4 reported immediately failed by osd.2 2023-07-10T17:13:44.245 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.687850+0000 mon.a (mon.0) 547 : cluster 0 osd.4 reported immediately failed by osd.6 2023-07-10T17:13:44.245 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.688240+0000 mon.a (mon.0) 548 : cluster 0 osd.4 reported immediately failed by osd.6 2023-07-10T17:13:44.245 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.713896+0000 mon.a (mon.0) 549 : cluster 0 osd.4 reported immediately failed by osd.0 2023-07-10T17:13:44.245 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.716676+0000 mon.a (mon.0) 550 : cluster 0 osd.4 reported immediately failed by osd.2 2023-07-10T17:13:44.245 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.717034+0000 mon.a (mon.0) 551 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.245 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.730380+0000 mon.a (mon.0) 552 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.245 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.736915+0000 mon.a (mon.0) 553 : cluster 0 osd.4 reported immediately failed by osd.5 2023-07-10T17:13:44.245 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.740986+0000 mon.a (mon.0) 554 : cluster 0 osd.4 reported immediately failed by osd.7 2023-07-10T17:13:44.246 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.920423+0000 mon.a (mon.0) 555 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.246 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.920778+0000 mon.a (mon.0) 556 : cluster 0 osd.4 reported immediately failed by osd.2 2023-07-10T17:13:44.246 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:43 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.931715+0000 mon.a (mon.0) 557 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.246 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:44 smithi192 bash[52403]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-osd.4 2023-07-10T17:13:44.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:42.529043+0000 mgr.y (mgr.25390) 231 : cluster 0 pgmap v152: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:13:44.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.482241+0000 mon.a (mon.0) 525 : cluster 0 osd.4 reported immediately failed by osd.7 2023-07-10T17:13:44.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.482338+0000 mon.a (mon.0) 526 : cluster 1 osd.4 failed (root=default,host=smithi192) (connection refused reported by osd.7) 2023-07-10T17:13:44.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.482851+0000 mon.a (mon.0) 527 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.483195+0000 mon.a (mon.0) 528 : cluster 0 osd.4 reported immediately failed by osd.0 2023-07-10T17:13:44.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.483658+0000 mon.a (mon.0) 529 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.483939+0000 mon.a (mon.0) 530 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.484373+0000 mon.a (mon.0) 531 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.531619+0000 mon.a (mon.0) 532 : cluster 0 osd.4 reported immediately failed by osd.6 2023-07-10T17:13:44.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.536653+0000 mon.a (mon.0) 533 : cluster 0 osd.4 reported immediately failed by osd.5 2023-07-10T17:13:44.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.539144+0000 mon.a (mon.0) 534 : cluster 0 osd.4 reported immediately failed by osd.7 2023-07-10T17:13:44.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.684249+0000 mon.a (mon.0) 535 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.684567+0000 mon.a (mon.0) 536 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.684901+0000 mon.a (mon.0) 537 : cluster 0 osd.4 reported immediately failed by osd.0 2023-07-10T17:13:44.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.685186+0000 mon.a (mon.0) 538 : cluster 0 osd.4 reported immediately failed by osd.0 2023-07-10T17:13:44.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.685697+0000 mon.a (mon.0) 539 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.685976+0000 mon.a (mon.0) 540 : cluster 0 osd.4 reported immediately failed by osd.7 2023-07-10T17:13:44.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.686167+0000 mon.a (mon.0) 541 : cluster 0 osd.4 reported immediately failed by osd.5 2023-07-10T17:13:44.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.686412+0000 mon.a (mon.0) 542 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.686690+0000 mon.a (mon.0) 543 : cluster 0 osd.4 reported immediately failed by osd.7 2023-07-10T17:13:44.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.686883+0000 mon.a (mon.0) 544 : cluster 0 osd.4 reported immediately failed by osd.5 2023-07-10T17:13:44.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.687130+0000 mon.a (mon.0) 545 : cluster 0 osd.4 reported immediately failed by osd.2 2023-07-10T17:13:44.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.687494+0000 mon.a (mon.0) 546 : cluster 0 osd.4 reported immediately failed by osd.2 2023-07-10T17:13:44.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.687850+0000 mon.a (mon.0) 547 : cluster 0 osd.4 reported immediately failed by osd.6 2023-07-10T17:13:44.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.688240+0000 mon.a (mon.0) 548 : cluster 0 osd.4 reported immediately failed by osd.6 2023-07-10T17:13:44.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.713896+0000 mon.a (mon.0) 549 : cluster 0 osd.4 reported immediately failed by osd.0 2023-07-10T17:13:44.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:42.529043+0000 mgr.y (mgr.25390) 231 : cluster 0 pgmap v152: 161 pgs: 161 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:13:44.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.482241+0000 mon.a (mon.0) 525 : cluster 0 osd.4 reported immediately failed by osd.7 2023-07-10T17:13:44.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.482338+0000 mon.a (mon.0) 526 : cluster 1 osd.4 failed (root=default,host=smithi192) (connection refused reported by osd.7) 2023-07-10T17:13:44.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.482851+0000 mon.a (mon.0) 527 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.483195+0000 mon.a (mon.0) 528 : cluster 0 osd.4 reported immediately failed by osd.0 2023-07-10T17:13:44.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.483658+0000 mon.a (mon.0) 529 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.483939+0000 mon.a (mon.0) 530 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.484373+0000 mon.a (mon.0) 531 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.531619+0000 mon.a (mon.0) 532 : cluster 0 osd.4 reported immediately failed by osd.6 2023-07-10T17:13:44.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.536653+0000 mon.a (mon.0) 533 : cluster 0 osd.4 reported immediately failed by osd.5 2023-07-10T17:13:44.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.539144+0000 mon.a (mon.0) 534 : cluster 0 osd.4 reported immediately failed by osd.7 2023-07-10T17:13:44.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.684249+0000 mon.a (mon.0) 535 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.684567+0000 mon.a (mon.0) 536 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.684901+0000 mon.a (mon.0) 537 : cluster 0 osd.4 reported immediately failed by osd.0 2023-07-10T17:13:44.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.685186+0000 mon.a (mon.0) 538 : cluster 0 osd.4 reported immediately failed by osd.0 2023-07-10T17:13:44.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.685697+0000 mon.a (mon.0) 539 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.685976+0000 mon.a (mon.0) 540 : cluster 0 osd.4 reported immediately failed by osd.7 2023-07-10T17:13:44.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.686167+0000 mon.a (mon.0) 541 : cluster 0 osd.4 reported immediately failed by osd.5 2023-07-10T17:13:44.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.686412+0000 mon.a (mon.0) 542 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.686690+0000 mon.a (mon.0) 543 : cluster 0 osd.4 reported immediately failed by osd.7 2023-07-10T17:13:44.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.686883+0000 mon.a (mon.0) 544 : cluster 0 osd.4 reported immediately failed by osd.5 2023-07-10T17:13:44.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.687130+0000 mon.a (mon.0) 545 : cluster 0 osd.4 reported immediately failed by osd.2 2023-07-10T17:13:44.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.687494+0000 mon.a (mon.0) 546 : cluster 0 osd.4 reported immediately failed by osd.2 2023-07-10T17:13:44.402 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.687850+0000 mon.a (mon.0) 547 : cluster 0 osd.4 reported immediately failed by osd.6 2023-07-10T17:13:44.402 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.688240+0000 mon.a (mon.0) 548 : cluster 0 osd.4 reported immediately failed by osd.6 2023-07-10T17:13:44.402 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.713896+0000 mon.a (mon.0) 549 : cluster 0 osd.4 reported immediately failed by osd.0 2023-07-10T17:13:44.402 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.716676+0000 mon.a (mon.0) 550 : cluster 0 osd.4 reported immediately failed by osd.2 2023-07-10T17:13:44.402 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.717034+0000 mon.a (mon.0) 551 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.402 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.730380+0000 mon.a (mon.0) 552 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.403 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.736915+0000 mon.a (mon.0) 553 : cluster 0 osd.4 reported immediately failed by osd.5 2023-07-10T17:13:44.403 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.740986+0000 mon.a (mon.0) 554 : cluster 0 osd.4 reported immediately failed by osd.7 2023-07-10T17:13:44.403 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.920423+0000 mon.a (mon.0) 555 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.403 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.920778+0000 mon.a (mon.0) 556 : cluster 0 osd.4 reported immediately failed by osd.2 2023-07-10T17:13:44.403 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.931715+0000 mon.a (mon.0) 557 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.403 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.716676+0000 mon.a (mon.0) 550 : cluster 0 osd.4 reported immediately failed by osd.2 2023-07-10T17:13:44.404 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.717034+0000 mon.a (mon.0) 551 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.404 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.730380+0000 mon.a (mon.0) 552 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.404 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.736915+0000 mon.a (mon.0) 553 : cluster 0 osd.4 reported immediately failed by osd.5 2023-07-10T17:13:44.404 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.740986+0000 mon.a (mon.0) 554 : cluster 0 osd.4 reported immediately failed by osd.7 2023-07-10T17:13:44.404 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.920423+0000 mon.a (mon.0) 555 : cluster 0 osd.4 reported immediately failed by osd.1 2023-07-10T17:13:44.404 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.920778+0000 mon.a (mon.0) 556 : cluster 0 osd.4 reported immediately failed by osd.2 2023-07-10T17:13:44.404 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:43 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.931715+0000 mon.a (mon.0) 557 : cluster 0 osd.4 reported immediately failed by osd.3 2023-07-10T17:13:44.577 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:13:44.577 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (6m) 18s ago 13m 10.9M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:13:44.577 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (5m) 2m ago 12m 25.6M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:13:44.578 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 error 18s ago 10m - - 2023-07-10T17:13:44.578 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (4m) 2m ago 17m 428M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:13:44.578 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (5m) 18s ago 19m 475M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:13:44.578 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (4m) 18s ago 19m 59.6M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:13:44.579 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (3m) 2m ago 17m 40.2M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 74a082a2e795 2023-07-10T17:13:44.579 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (3m) 18s ago 17m 49.9M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:13:44.579 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (7m) 18s ago 13m 9371k - 0.18.1 e5a616e4b9cf 3bd058d91353 2023-07-10T17:13:44.579 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (6m) 2m ago 13m 3607k - 0.18.1 e5a616e4b9cf 63551615c679 2023-07-10T17:13:44.579 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (113s) 18s ago 17m 53.2M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 0562f587fbca 2023-07-10T17:13:44.579 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (83s) 18s ago 16m 51.6M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 5b2a12a0bfc9 2023-07-10T17:13:44.580 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (53s) 18s ago 16m 47.9M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 88096bc0faa9 2023-07-10T17:13:44.580 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (24s) 18s ago 15m - 4096M 18.1.2-30-g8507faa7 17e8f0d79247 61ae280c471b 2023-07-10T17:13:44.580 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (15m) 2m ago 15m 55.1M 4096M 16.2.0 24ecd6d5f14c 75fc0b36aab8 2023-07-10T17:13:44.580 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (15m) 2m ago 15m 52.8M 4096M 16.2.0 24ecd6d5f14c d673065333dd 2023-07-10T17:13:44.581 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (14m) 2m ago 14m 53.7M 4096M 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:13:44.581 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (14m) 2m ago 14m 56.7M 4096M 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:13:44.581 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 error 2m ago 13m - - 2023-07-10T17:13:44.581 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (11m) 18s ago 11m 55.9M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:13:44.582 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (11m) 2m ago 11m 54.8M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:13:44.582 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (10m) 18s ago 10m 55.5M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:13:44.582 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (10m) 2m ago 10m 56.6M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:13:45.023 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:13:45.023 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:13:45.023 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:13:45.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:13:45.024 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:13:45.024 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:13:45.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:13:45.024 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:13:45.024 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3, 2023-07-10T17:13:45.024 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 4 2023-07-10T17:13:45.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:13:45.025 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:13:45.025 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:13:45.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:13:45.025 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:13:45.025 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 7, 2023-07-10T17:13:45.025 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 9 2023-07-10T17:13:45.025 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:13:45.025 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:13:45.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:44 smithi192 bash[50362]: audit 2023-07-10T17:13:43.806945+0000 mgr.y (mgr.25390) 232 : audit 0 from='client.34581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:45.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:44 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.974591+0000 mon.a (mon.0) 558 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:13:45.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:44 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.989916+0000 osd.6 (osd.6) 29 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:44 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.992181+0000 osd.6 (osd.6) 30 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:44 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.993687+0000 mon.a (mon.0) 559 : cluster 0 osdmap e98: 8 total, 7 up, 8 in 2023-07-10T17:13:45.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:44 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.994609+0000 osd.5 (osd.5) 30 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:44 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.995799+0000 osd.6 (osd.6) 31 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:44 smithi192 bash[50362]: cluster 2023-07-10T17:13:43.996827+0000 osd.7 (osd.7) 30 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:44 smithi192 bash[50362]: cluster 2023-07-10T17:13:44.001086+0000 osd.5 (osd.5) 31 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:44 smithi192 bash[50362]: audit 2023-07-10T17:13:44.190287+0000 mgr.y (mgr.25390) 233 : audit 0 from='client.44443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:45.393 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:13:45.393 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:13:45.393 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:13:45.393 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:13:45.393 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:13:45.394 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2023-07-10T17:13:45.394 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2023-07-10T17:13:45.394 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:13:45.394 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "9/23 daemons upgraded", 2023-07-10T17:13:45.394 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading osd daemons", 2023-07-10T17:13:45.394 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:13:45.395 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:13:45.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[63900]: audit 2023-07-10T17:13:43.806945+0000 mgr.y (mgr.25390) 232 : audit 0 from='client.34581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:45.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.974591+0000 mon.a (mon.0) 558 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:13:45.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.989916+0000 osd.6 (osd.6) 29 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.992181+0000 osd.6 (osd.6) 30 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.993687+0000 mon.a (mon.0) 559 : cluster 0 osdmap e98: 8 total, 7 up, 8 in 2023-07-10T17:13:45.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.994609+0000 osd.5 (osd.5) 30 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.995799+0000 osd.6 (osd.6) 31 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[63900]: cluster 2023-07-10T17:13:43.996827+0000 osd.7 (osd.7) 30 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[63900]: cluster 2023-07-10T17:13:44.001086+0000 osd.5 (osd.5) 31 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[63900]: audit 2023-07-10T17:13:44.190287+0000 mgr.y (mgr.25390) 233 : audit 0 from='client.44443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:45.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[65980]: audit 2023-07-10T17:13:43.806945+0000 mgr.y (mgr.25390) 232 : audit 0 from='client.34581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:45.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.974591+0000 mon.a (mon.0) 558 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:13:45.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.989916+0000 osd.6 (osd.6) 29 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.992181+0000 osd.6 (osd.6) 30 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.993687+0000 mon.a (mon.0) 559 : cluster 0 osdmap e98: 8 total, 7 up, 8 in 2023-07-10T17:13:45.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.994609+0000 osd.5 (osd.5) 30 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.995799+0000 osd.6 (osd.6) 31 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[65980]: cluster 2023-07-10T17:13:43.996827+0000 osd.7 (osd.7) 30 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[65980]: cluster 2023-07-10T17:13:44.001086+0000 osd.5 (osd.5) 31 : cluster 3 failed to encode map e98 with expected crc 2023-07-10T17:13:45.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:44 smithi116 bash[65980]: audit 2023-07-10T17:13:44.190287+0000 mgr.y (mgr.25390) 233 : audit 0 from='client.44443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:45.825 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 2 failed cephadm daemon(s); 1 osds down 2023-07-10T17:13:45.826 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:13:45.826 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:13:45.826 INFO:teuthology.orchestra.run.smithi116.stdout: daemon prometheus.a on smithi192 is in error state 2023-07-10T17:13:45.826 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] OSD_DOWN: 1 osds down 2023-07-10T17:13:45.826 INFO:teuthology.orchestra.run.smithi116.stdout: osd.4 (root=default,host=smithi192) is down 2023-07-10T17:13:46.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:45 smithi192 bash[50362]: cluster 2023-07-10T17:13:44.529763+0000 mgr.y (mgr.25390) 234 : cluster 0 pgmap v154: 161 pgs: 23 stale+active+clean, 138 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:13:46.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:45 smithi192 bash[50362]: audit 2023-07-10T17:13:44.546504+0000 mgr.y (mgr.25390) 235 : audit 0 from='client.44449 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:46.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:45 smithi192 bash[50362]: cluster 2023-07-10T17:13:44.994287+0000 mon.a (mon.0) 560 : cluster 0 osdmap e99: 8 total, 7 up, 8 in 2023-07-10T17:13:46.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:45 smithi192 bash[50362]: cluster 2023-07-10T17:13:44.996057+0000 osd.7 (osd.7) 31 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:45 smithi192 bash[50362]: cluster 2023-07-10T17:13:44.996117+0000 osd.5 (osd.5) 32 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:45 smithi192 bash[50362]: cluster 2023-07-10T17:13:44.998817+0000 osd.6 (osd.6) 32 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:45 smithi192 bash[50362]: cluster 2023-07-10T17:13:44.999865+0000 osd.7 (osd.7) 32 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:45 smithi192 bash[50362]: cluster 2023-07-10T17:13:45.001800+0000 osd.5 (osd.5) 33 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:45 smithi192 bash[50362]: cluster 2023-07-10T17:13:45.002545+0000 osd.6 (osd.6) 33 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:45 smithi192 bash[50362]: cluster 2023-07-10T17:13:45.006639+0000 osd.6 (osd.6) 34 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:45 smithi192 bash[50362]: cluster 2023-07-10T17:13:45.009959+0000 osd.6 (osd.6) 35 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:45 smithi192 bash[50362]: audit 2023-07-10T17:13:45.018577+0000 mon.a (mon.0) 561 : audit 0 from='client.34599 172.21.15.116:0/1159976653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:46.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:45 smithi192 bash[50362]: audit 2023-07-10T17:13:45.821278+0000 mon.c (mon.1) 21 : audit 0 from='client.34611 172.21.15.116:0/621237350' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:13:46.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[63900]: cluster 2023-07-10T17:13:44.529763+0000 mgr.y (mgr.25390) 234 : cluster 0 pgmap v154: 161 pgs: 23 stale+active+clean, 138 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:13:46.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[63900]: audit 2023-07-10T17:13:44.546504+0000 mgr.y (mgr.25390) 235 : audit 0 from='client.44449 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:46.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[63900]: cluster 2023-07-10T17:13:44.994287+0000 mon.a (mon.0) 560 : cluster 0 osdmap e99: 8 total, 7 up, 8 in 2023-07-10T17:13:46.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[63900]: cluster 2023-07-10T17:13:44.996057+0000 osd.7 (osd.7) 31 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[63900]: cluster 2023-07-10T17:13:44.996117+0000 osd.5 (osd.5) 32 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[63900]: cluster 2023-07-10T17:13:44.998817+0000 osd.6 (osd.6) 32 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[63900]: cluster 2023-07-10T17:13:44.999865+0000 osd.7 (osd.7) 32 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[63900]: cluster 2023-07-10T17:13:45.001800+0000 osd.5 (osd.5) 33 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[63900]: cluster 2023-07-10T17:13:45.002545+0000 osd.6 (osd.6) 33 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[63900]: cluster 2023-07-10T17:13:45.006639+0000 osd.6 (osd.6) 34 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[63900]: cluster 2023-07-10T17:13:45.009959+0000 osd.6 (osd.6) 35 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[63900]: audit 2023-07-10T17:13:45.018577+0000 mon.a (mon.0) 561 : audit 0 from='client.34599 172.21.15.116:0/1159976653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:46.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[63900]: audit 2023-07-10T17:13:45.821278+0000 mon.c (mon.1) 21 : audit 0 from='client.34611 172.21.15.116:0/621237350' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:13:46.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[65980]: cluster 2023-07-10T17:13:44.529763+0000 mgr.y (mgr.25390) 234 : cluster 0 pgmap v154: 161 pgs: 23 stale+active+clean, 138 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:13:46.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[65980]: audit 2023-07-10T17:13:44.546504+0000 mgr.y (mgr.25390) 235 : audit 0 from='client.44449 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:46.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[65980]: cluster 2023-07-10T17:13:44.994287+0000 mon.a (mon.0) 560 : cluster 0 osdmap e99: 8 total, 7 up, 8 in 2023-07-10T17:13:46.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[65980]: cluster 2023-07-10T17:13:44.996057+0000 osd.7 (osd.7) 31 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[65980]: cluster 2023-07-10T17:13:44.996117+0000 osd.5 (osd.5) 32 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[65980]: cluster 2023-07-10T17:13:44.998817+0000 osd.6 (osd.6) 32 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[65980]: cluster 2023-07-10T17:13:44.999865+0000 osd.7 (osd.7) 32 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[65980]: cluster 2023-07-10T17:13:45.001800+0000 osd.5 (osd.5) 33 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[65980]: cluster 2023-07-10T17:13:45.002545+0000 osd.6 (osd.6) 33 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[65980]: cluster 2023-07-10T17:13:45.006639+0000 osd.6 (osd.6) 34 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[65980]: cluster 2023-07-10T17:13:45.009959+0000 osd.6 (osd.6) 35 : cluster 3 failed to encode map e99 with expected crc 2023-07-10T17:13:46.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[65980]: audit 2023-07-10T17:13:45.018577+0000 mon.a (mon.0) 561 : audit 0 from='client.34599 172.21.15.116:0/1159976653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:13:46.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:45 smithi116 bash[65980]: audit 2023-07-10T17:13:45.821278+0000 mon.c (mon.1) 21 : audit 0 from='client.34611 172.21.15.116:0/621237350' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:13:46.868 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:46 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.4.service: Succeeded. 2023-07-10T17:13:46.868 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:46 smithi192 systemd[1]: Stopped Ceph osd.4 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:13:46.869 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:46 smithi192 systemd[1]: Started Ceph osd.4 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:13:47.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:47 smithi192 bash[50362]: audit 2023-07-10T17:13:45.387918+0000 mgr.y (mgr.25390) 236 : audit 0 from='client.34605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:47.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:47 smithi192 bash[50362]: audit 2023-07-10T17:13:46.822377+0000 mon.a (mon.0) 562 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:47.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:47 smithi192 bash[50362]: audit 2023-07-10T17:13:46.833972+0000 mon.a (mon.0) 563 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:47.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:47 smithi116 bash[63900]: audit 2023-07-10T17:13:45.387918+0000 mgr.y (mgr.25390) 236 : audit 0 from='client.34605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:47.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:47 smithi116 bash[63900]: audit 2023-07-10T17:13:46.822377+0000 mon.a (mon.0) 562 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:47.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:47 smithi116 bash[63900]: audit 2023-07-10T17:13:46.833972+0000 mon.a (mon.0) 563 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:47.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:47 smithi116 bash[65980]: audit 2023-07-10T17:13:45.387918+0000 mgr.y (mgr.25390) 236 : audit 0 from='client.34605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:13:47.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:47 smithi116 bash[65980]: audit 2023-07-10T17:13:46.822377+0000 mon.a (mon.0) 562 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:47.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:47 smithi116 bash[65980]: audit 2023-07-10T17:13:46.833972+0000 mon.a (mon.0) 563 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:48.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:48 smithi192 bash[50362]: cluster 2023-07-10T17:13:46.531098+0000 mgr.y (mgr.25390) 237 : cluster 0 pgmap v156: 161 pgs: 7 active+undersized+degraded, 20 active+undersized, 11 stale+active+clean, 123 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 40/999 objects degraded (4.004%) 2023-07-10T17:13:48.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:48 smithi192 bash[50362]: cluster 2023-07-10T17:13:46.992256+0000 mon.a (mon.0) 564 : cluster 3 Health check failed: Degraded data redundancy: 40/999 objects degraded (4.004%), 7 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:48.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:48 smithi116 bash[63900]: cluster 2023-07-10T17:13:46.531098+0000 mgr.y (mgr.25390) 237 : cluster 0 pgmap v156: 161 pgs: 7 active+undersized+degraded, 20 active+undersized, 11 stale+active+clean, 123 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 40/999 objects degraded (4.004%) 2023-07-10T17:13:48.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:48 smithi116 bash[63900]: cluster 2023-07-10T17:13:46.992256+0000 mon.a (mon.0) 564 : cluster 3 Health check failed: Degraded data redundancy: 40/999 objects degraded (4.004%), 7 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:48.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:48 smithi116 bash[65980]: cluster 2023-07-10T17:13:46.531098+0000 mgr.y (mgr.25390) 237 : cluster 0 pgmap v156: 161 pgs: 7 active+undersized+degraded, 20 active+undersized, 11 stale+active+clean, 123 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 40/999 objects degraded (4.004%) 2023-07-10T17:13:48.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:48 smithi116 bash[65980]: cluster 2023-07-10T17:13:46.992256+0000 mon.a (mon.0) 564 : cluster 3 Health check failed: Degraded data redundancy: 40/999 objects degraded (4.004%), 7 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:48.797 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:48 smithi192 bash[52666]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-07-10T17:13:48.797 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:48 smithi192 bash[52666]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2023-07-10T17:13:48.797 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:48 smithi192 bash[52666]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2023-07-10T17:13:48.798 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:48 smithi192 bash[52666]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-07-10T17:13:48.798 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:48 smithi192 bash[52666]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2023-07-10T17:13:48.798 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:48 smithi192 bash[52666]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-07-10T17:13:48.798 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:48 smithi192 bash[52666]: --> ceph-volume raw activate successful for osd ID: 4 2023-07-10T17:13:50.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:50 smithi192 bash[50362]: cluster 2023-07-10T17:13:48.532762+0000 mgr.y (mgr.25390) 238 : cluster 0 pgmap v157: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2023-07-10T17:13:50.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:50 smithi116 bash[63900]: cluster 2023-07-10T17:13:48.532762+0000 mgr.y (mgr.25390) 238 : cluster 0 pgmap v157: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2023-07-10T17:13:50.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:50 smithi116 bash[65980]: cluster 2023-07-10T17:13:48.532762+0000 mgr.y (mgr.25390) 238 : cluster 0 pgmap v157: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2023-07-10T17:13:50.637 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:50 smithi192 bash[52978]: debug 2023-07-10T17:13:50.313+0000 7f06fb2b5700 1 -- 172.21.15.192:0/2091954179 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560738efc4e0 con 0x560738f57000 2023-07-10T17:13:51.137 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:50 smithi192 bash[52978]: debug 2023-07-10T17:13:50.849+0000 7f070443a640 -1 Falling back to public interface 2023-07-10T17:13:52.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:52 smithi192 bash[50362]: cluster 2023-07-10T17:13:50.533855+0000 mgr.y (mgr.25390) 239 : cluster 0 pgmap v158: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2023-07-10T17:13:52.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:52 smithi116 bash[63900]: cluster 2023-07-10T17:13:50.533855+0000 mgr.y (mgr.25390) 239 : cluster 0 pgmap v158: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2023-07-10T17:13:52.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:52 smithi116 bash[65980]: cluster 2023-07-10T17:13:50.533855+0000 mgr.y (mgr.25390) 239 : cluster 0 pgmap v158: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2023-07-10T17:13:53.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:53 smithi192 bash[50362]: cluster 2023-07-10T17:13:52.567030+0000 mon.a (mon.0) 565 : cluster 3 Health check update: Degraded data redundancy: 164/999 objects degraded (16.416%), 25 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:53.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:53 smithi116 bash[63900]: cluster 2023-07-10T17:13:52.567030+0000 mon.a (mon.0) 565 : cluster 3 Health check update: Degraded data redundancy: 164/999 objects degraded (16.416%), 25 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:53.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:53 smithi116 bash[65980]: cluster 2023-07-10T17:13:52.567030+0000 mon.a (mon.0) 565 : cluster 3 Health check update: Degraded data redundancy: 164/999 objects degraded (16.416%), 25 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:54.387 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:53 smithi192 bash[52978]: debug 2023-07-10T17:13:53.929+0000 7f070443a640 -1 osd.4 97 log_to_monitors true 2023-07-10T17:13:54.387 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:13:54 smithi192 bash[52978]: debug 2023-07-10T17:13:54.125+0000 7f06f48a5700 -1 osd.4 97 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-10T17:13:54.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:54 smithi192 bash[50362]: cluster 2023-07-10T17:13:52.535805+0000 mgr.y (mgr.25390) 240 : cluster 0 pgmap v159: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2023-07-10T17:13:54.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:54 smithi192 bash[50362]: audit 2023-07-10T17:13:53.940510+0000 mon.a (mon.0) 566 : audit 1 from='osd.4 [v2:172.21.15.192:6800/4143205849,v1:172.21.15.192:6801/4143205849]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-10T17:13:54.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:54 smithi116 bash[63900]: cluster 2023-07-10T17:13:52.535805+0000 mgr.y (mgr.25390) 240 : cluster 0 pgmap v159: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2023-07-10T17:13:54.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:54 smithi116 bash[63900]: audit 2023-07-10T17:13:53.940510+0000 mon.a (mon.0) 566 : audit 1 from='osd.4 [v2:172.21.15.192:6800/4143205849,v1:172.21.15.192:6801/4143205849]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-10T17:13:54.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:54 smithi116 bash[65980]: cluster 2023-07-10T17:13:52.535805+0000 mgr.y (mgr.25390) 240 : cluster 0 pgmap v159: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 604 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2023-07-10T17:13:54.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:54 smithi116 bash[65980]: audit 2023-07-10T17:13:53.940510+0000 mon.a (mon.0) 566 : audit 1 from='osd.4 [v2:172.21.15.192:6800/4143205849,v1:172.21.15.192:6801/4143205849]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-10T17:13:55.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:55 smithi192 bash[50362]: cluster 2023-07-10T17:13:54.044280+0000 osd.7 (osd.7) 33 : cluster 3 failed to encode map e100 with expected crc 2023-07-10T17:13:55.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:55 smithi192 bash[50362]: cluster 2023-07-10T17:13:54.044620+0000 osd.6 (osd.6) 36 : cluster 3 failed to encode map e100 with expected crc 2023-07-10T17:13:55.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:55 smithi192 bash[50362]: audit 2023-07-10T17:13:54.047401+0000 mon.a (mon.0) 567 : audit 1 from='osd.4 [v2:172.21.15.192:6800/4143205849,v1:172.21.15.192:6801/4143205849]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-10T17:13:55.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:55 smithi192 bash[50362]: cluster 2023-07-10T17:13:54.047629+0000 mon.a (mon.0) 568 : cluster 0 osdmap e100: 8 total, 7 up, 8 in 2023-07-10T17:13:55.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:55 smithi192 bash[50362]: audit 2023-07-10T17:13:54.048458+0000 mon.a (mon.0) 569 : audit 1 from='osd.4 [v2:172.21.15.192:6800/4143205849,v1:172.21.15.192:6801/4143205849]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:13:55.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:55 smithi192 bash[50362]: audit 2023-07-10T17:13:54.432322+0000 mon.a (mon.0) 570 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:55.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:55 smithi192 bash[50362]: audit 2023-07-10T17:13:54.443038+0000 mon.a (mon.0) 571 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:55.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:55 smithi116 bash[63900]: cluster 2023-07-10T17:13:54.044280+0000 osd.7 (osd.7) 33 : cluster 3 failed to encode map e100 with expected crc 2023-07-10T17:13:55.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:55 smithi116 bash[63900]: cluster 2023-07-10T17:13:54.044620+0000 osd.6 (osd.6) 36 : cluster 3 failed to encode map e100 with expected crc 2023-07-10T17:13:55.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:55 smithi116 bash[63900]: audit 2023-07-10T17:13:54.047401+0000 mon.a (mon.0) 567 : audit 1 from='osd.4 [v2:172.21.15.192:6800/4143205849,v1:172.21.15.192:6801/4143205849]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-10T17:13:55.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:55 smithi116 bash[63900]: cluster 2023-07-10T17:13:54.047629+0000 mon.a (mon.0) 568 : cluster 0 osdmap e100: 8 total, 7 up, 8 in 2023-07-10T17:13:55.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:55 smithi116 bash[63900]: audit 2023-07-10T17:13:54.048458+0000 mon.a (mon.0) 569 : audit 1 from='osd.4 [v2:172.21.15.192:6800/4143205849,v1:172.21.15.192:6801/4143205849]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:13:55.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:55 smithi116 bash[63900]: audit 2023-07-10T17:13:54.432322+0000 mon.a (mon.0) 570 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:55.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:55 smithi116 bash[63900]: audit 2023-07-10T17:13:54.443038+0000 mon.a (mon.0) 571 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:55.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:55 smithi116 bash[65980]: cluster 2023-07-10T17:13:54.044280+0000 osd.7 (osd.7) 33 : cluster 3 failed to encode map e100 with expected crc 2023-07-10T17:13:55.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:55 smithi116 bash[65980]: cluster 2023-07-10T17:13:54.044620+0000 osd.6 (osd.6) 36 : cluster 3 failed to encode map e100 with expected crc 2023-07-10T17:13:55.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:55 smithi116 bash[65980]: audit 2023-07-10T17:13:54.047401+0000 mon.a (mon.0) 567 : audit 1 from='osd.4 [v2:172.21.15.192:6800/4143205849,v1:172.21.15.192:6801/4143205849]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-10T17:13:55.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:55 smithi116 bash[65980]: cluster 2023-07-10T17:13:54.047629+0000 mon.a (mon.0) 568 : cluster 0 osdmap e100: 8 total, 7 up, 8 in 2023-07-10T17:13:55.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:55 smithi116 bash[65980]: audit 2023-07-10T17:13:54.048458+0000 mon.a (mon.0) 569 : audit 1 from='osd.4 [v2:172.21.15.192:6800/4143205849,v1:172.21.15.192:6801/4143205849]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:13:55.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:55 smithi116 bash[65980]: audit 2023-07-10T17:13:54.432322+0000 mon.a (mon.0) 570 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:55.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:55 smithi116 bash[65980]: audit 2023-07-10T17:13:54.443038+0000 mon.a (mon.0) 571 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:56.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:56 smithi192 bash[50362]: cluster 2023-07-10T17:13:54.536881+0000 mgr.y (mgr.25390) 241 : cluster 0 pgmap v161: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 604 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2023-07-10T17:13:56.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:56 smithi192 bash[50362]: cluster 2023-07-10T17:13:55.040514+0000 mon.a (mon.0) 572 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:13:56.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:56 smithi192 bash[50362]: cluster 2023-07-10T17:13:55.045340+0000 osd.5 (osd.5) 34 : cluster 3 failed to encode map e101 with expected crc 2023-07-10T17:13:56.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:56 smithi192 bash[50362]: cluster 2023-07-10T17:13:55.049562+0000 mon.a (mon.0) 573 : cluster 1 osd.4 [v2:172.21.15.192:6800/4143205849,v1:172.21.15.192:6801/4143205849] boot 2023-07-10T17:13:56.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:56 smithi192 bash[50362]: cluster 2023-07-10T17:13:55.049677+0000 mon.a (mon.0) 574 : cluster 0 osdmap e101: 8 total, 8 up, 8 in 2023-07-10T17:13:56.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:56 smithi192 bash[50362]: audit 2023-07-10T17:13:55.050539+0000 mon.a (mon.0) 575 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T17:13:56.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:56 smithi192 bash[50362]: cluster 2023-07-10T17:13:55.054217+0000 osd.7 (osd.7) 34 : cluster 3 failed to encode map e101 with expected crc 2023-07-10T17:13:56.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:56 smithi192 bash[50362]: audit 2023-07-10T17:13:55.240891+0000 mon.a (mon.0) 576 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:56.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:56 smithi192 bash[50362]: audit 2023-07-10T17:13:55.253038+0000 mon.a (mon.0) 577 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:56.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[65980]: cluster 2023-07-10T17:13:54.536881+0000 mgr.y (mgr.25390) 241 : cluster 0 pgmap v161: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 604 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2023-07-10T17:13:56.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[65980]: cluster 2023-07-10T17:13:55.040514+0000 mon.a (mon.0) 572 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:13:56.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[65980]: cluster 2023-07-10T17:13:55.045340+0000 osd.5 (osd.5) 34 : cluster 3 failed to encode map e101 with expected crc 2023-07-10T17:13:56.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[65980]: cluster 2023-07-10T17:13:55.049562+0000 mon.a (mon.0) 573 : cluster 1 osd.4 [v2:172.21.15.192:6800/4143205849,v1:172.21.15.192:6801/4143205849] boot 2023-07-10T17:13:56.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[65980]: cluster 2023-07-10T17:13:55.049677+0000 mon.a (mon.0) 574 : cluster 0 osdmap e101: 8 total, 8 up, 8 in 2023-07-10T17:13:56.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[65980]: audit 2023-07-10T17:13:55.050539+0000 mon.a (mon.0) 575 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T17:13:56.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[65980]: cluster 2023-07-10T17:13:55.054217+0000 osd.7 (osd.7) 34 : cluster 3 failed to encode map e101 with expected crc 2023-07-10T17:13:56.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[65980]: audit 2023-07-10T17:13:55.240891+0000 mon.a (mon.0) 576 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:56.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[65980]: audit 2023-07-10T17:13:55.253038+0000 mon.a (mon.0) 577 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:56.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[63900]: cluster 2023-07-10T17:13:54.536881+0000 mgr.y (mgr.25390) 241 : cluster 0 pgmap v161: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 604 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2023-07-10T17:13:56.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[63900]: cluster 2023-07-10T17:13:55.040514+0000 mon.a (mon.0) 572 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:13:56.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[63900]: cluster 2023-07-10T17:13:55.045340+0000 osd.5 (osd.5) 34 : cluster 3 failed to encode map e101 with expected crc 2023-07-10T17:13:56.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[63900]: cluster 2023-07-10T17:13:55.049562+0000 mon.a (mon.0) 573 : cluster 1 osd.4 [v2:172.21.15.192:6800/4143205849,v1:172.21.15.192:6801/4143205849] boot 2023-07-10T17:13:56.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[63900]: cluster 2023-07-10T17:13:55.049677+0000 mon.a (mon.0) 574 : cluster 0 osdmap e101: 8 total, 8 up, 8 in 2023-07-10T17:13:56.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[63900]: audit 2023-07-10T17:13:55.050539+0000 mon.a (mon.0) 575 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-10T17:13:56.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[63900]: cluster 2023-07-10T17:13:55.054217+0000 osd.7 (osd.7) 34 : cluster 3 failed to encode map e101 with expected crc 2023-07-10T17:13:56.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[63900]: audit 2023-07-10T17:13:55.240891+0000 mon.a (mon.0) 576 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:56.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:56 smithi116 bash[63900]: audit 2023-07-10T17:13:55.253038+0000 mon.a (mon.0) 577 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:13:57.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:57 smithi192 bash[50362]: cluster 2023-07-10T17:13:56.058356+0000 osd.7 (osd.7) 35 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:57 smithi192 bash[50362]: cluster 2023-07-10T17:13:56.062140+0000 mon.a (mon.0) 578 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2023-07-10T17:13:57.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:57 smithi192 bash[50362]: cluster 2023-07-10T17:13:56.063721+0000 osd.7 (osd.7) 36 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:57 smithi192 bash[50362]: cluster 2023-07-10T17:13:56.064304+0000 osd.5 (osd.5) 35 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:57 smithi192 bash[50362]: cluster 2023-07-10T17:13:56.064733+0000 osd.6 (osd.6) 37 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:57 smithi192 bash[50362]: cluster 2023-07-10T17:13:56.070679+0000 osd.5 (osd.5) 36 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:57 smithi192 bash[50362]: cluster 2023-07-10T17:13:56.073572+0000 osd.5 (osd.5) 37 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:57 smithi116 bash[65980]: cluster 2023-07-10T17:13:56.058356+0000 osd.7 (osd.7) 35 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:57 smithi116 bash[65980]: cluster 2023-07-10T17:13:56.062140+0000 mon.a (mon.0) 578 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2023-07-10T17:13:57.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:57 smithi116 bash[65980]: cluster 2023-07-10T17:13:56.063721+0000 osd.7 (osd.7) 36 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:57 smithi116 bash[65980]: cluster 2023-07-10T17:13:56.064304+0000 osd.5 (osd.5) 35 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:57 smithi116 bash[65980]: cluster 2023-07-10T17:13:56.064733+0000 osd.6 (osd.6) 37 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:57 smithi116 bash[65980]: cluster 2023-07-10T17:13:56.070679+0000 osd.5 (osd.5) 36 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:57 smithi116 bash[65980]: cluster 2023-07-10T17:13:56.073572+0000 osd.5 (osd.5) 37 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:57 smithi116 bash[63900]: cluster 2023-07-10T17:13:56.058356+0000 osd.7 (osd.7) 35 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:57 smithi116 bash[63900]: cluster 2023-07-10T17:13:56.062140+0000 mon.a (mon.0) 578 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2023-07-10T17:13:57.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:57 smithi116 bash[63900]: cluster 2023-07-10T17:13:56.063721+0000 osd.7 (osd.7) 36 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:57 smithi116 bash[63900]: cluster 2023-07-10T17:13:56.064304+0000 osd.5 (osd.5) 35 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:57 smithi116 bash[63900]: cluster 2023-07-10T17:13:56.064733+0000 osd.6 (osd.6) 37 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:57 smithi116 bash[63900]: cluster 2023-07-10T17:13:56.070679+0000 osd.5 (osd.5) 36 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:57.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:57 smithi116 bash[63900]: cluster 2023-07-10T17:13:56.073572+0000 osd.5 (osd.5) 37 : cluster 3 failed to encode map e102 with expected crc 2023-07-10T17:13:58.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:58 smithi192 bash[50362]: cluster 2023-07-10T17:13:56.538036+0000 mgr.y (mgr.25390) 242 : cluster 0 pgmap v164: 161 pgs: 3 activating, 12 peering, 22 active+undersized+degraded, 31 active+undersized, 93 active+clean; 604 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 142/999 objects degraded (14.214%) 2023-07-10T17:13:58.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:13:58 smithi192 bash[50362]: cluster 2023-07-10T17:13:57.568779+0000 mon.a (mon.0) 579 : cluster 3 Health check update: Degraded data redundancy: 142/999 objects degraded (14.214%), 22 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:58.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:58 smithi116 bash[63900]: cluster 2023-07-10T17:13:56.538036+0000 mgr.y (mgr.25390) 242 : cluster 0 pgmap v164: 161 pgs: 3 activating, 12 peering, 22 active+undersized+degraded, 31 active+undersized, 93 active+clean; 604 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 142/999 objects degraded (14.214%) 2023-07-10T17:13:58.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:13:58 smithi116 bash[63900]: cluster 2023-07-10T17:13:57.568779+0000 mon.a (mon.0) 579 : cluster 3 Health check update: Degraded data redundancy: 142/999 objects degraded (14.214%), 22 pgs degraded (PG_DEGRADED) 2023-07-10T17:13:58.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:58 smithi116 bash[65980]: cluster 2023-07-10T17:13:56.538036+0000 mgr.y (mgr.25390) 242 : cluster 0 pgmap v164: 161 pgs: 3 activating, 12 peering, 22 active+undersized+degraded, 31 active+undersized, 93 active+clean; 604 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 142/999 objects degraded (14.214%) 2023-07-10T17:13:58.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:13:58 smithi116 bash[65980]: cluster 2023-07-10T17:13:57.568779+0000 mon.a (mon.0) 579 : cluster 3 Health check update: Degraded data redundancy: 142/999 objects degraded (14.214%), 22 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:00.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:00 smithi192 bash[50362]: cluster 2023-07-10T17:13:58.539503+0000 mgr.y (mgr.25390) 243 : cluster 0 pgmap v165: 161 pgs: 3 activating, 12 peering, 9 active+undersized+degraded, 14 active+undersized, 123 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 41/999 objects degraded (4.104%) 2023-07-10T17:14:00.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:00 smithi116 bash[63900]: cluster 2023-07-10T17:13:58.539503+0000 mgr.y (mgr.25390) 243 : cluster 0 pgmap v165: 161 pgs: 3 activating, 12 peering, 9 active+undersized+degraded, 14 active+undersized, 123 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 41/999 objects degraded (4.104%) 2023-07-10T17:14:00.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:00 smithi116 bash[65980]: cluster 2023-07-10T17:13:58.539503+0000 mgr.y (mgr.25390) 243 : cluster 0 pgmap v165: 161 pgs: 3 activating, 12 peering, 9 active+undersized+degraded, 14 active+undersized, 123 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 41/999 objects degraded (4.104%) 2023-07-10T17:14:01.485 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:01 smithi192 bash[50362]: cluster 2023-07-10T17:14:01.084567+0000 mon.a (mon.0) 580 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/999 objects degraded (4.104%), 9 pgs degraded) 2023-07-10T17:14:01.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:01 smithi116 bash[65980]: cluster 2023-07-10T17:14:01.084567+0000 mon.a (mon.0) 580 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/999 objects degraded (4.104%), 9 pgs degraded) 2023-07-10T17:14:01.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:01 smithi116 bash[63900]: cluster 2023-07-10T17:14:01.084567+0000 mon.a (mon.0) 580 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/999 objects degraded (4.104%), 9 pgs degraded) 2023-07-10T17:14:02.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:02 smithi192 bash[50362]: cluster 2023-07-10T17:14:00.540450+0000 mgr.y (mgr.25390) 244 : cluster 0 pgmap v166: 161 pgs: 3 activating, 12 peering, 146 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:02.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:02 smithi116 bash[65980]: cluster 2023-07-10T17:14:00.540450+0000 mgr.y (mgr.25390) 244 : cluster 0 pgmap v166: 161 pgs: 3 activating, 12 peering, 146 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:02.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:02 smithi116 bash[63900]: cluster 2023-07-10T17:14:00.540450+0000 mgr.y (mgr.25390) 244 : cluster 0 pgmap v166: 161 pgs: 3 activating, 12 peering, 146 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:04.493 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:04 smithi192 bash[50362]: cluster 2023-07-10T17:14:02.542072+0000 mgr.y (mgr.25390) 245 : cluster 0 pgmap v167: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:04.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:04 smithi116 bash[65980]: cluster 2023-07-10T17:14:02.542072+0000 mgr.y (mgr.25390) 245 : cluster 0 pgmap v167: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:04.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:04 smithi116 bash[63900]: cluster 2023-07-10T17:14:02.542072+0000 mgr.y (mgr.25390) 245 : cluster 0 pgmap v167: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:06.341 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:06 smithi192 bash[50362]: cluster 2023-07-10T17:14:04.543041+0000 mgr.y (mgr.25390) 246 : cluster 0 pgmap v168: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:06.342 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:06 smithi192 bash[50362]: audit 2023-07-10T17:14:05.054691+0000 mon.a (mon.0) 581 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:06.342 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:06 smithi192 bash[50362]: audit 2023-07-10T17:14:05.065982+0000 mon.a (mon.0) 582 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:06.342 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:06 smithi192 bash[50362]: audit 2023-07-10T17:14:05.068042+0000 mon.a (mon.0) 583 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:14:06.342 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:06 smithi192 bash[50362]: audit 2023-07-10T17:14:05.069905+0000 mon.a (mon.0) 584 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:14:06.343 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:06 smithi192 bash[50362]: cephadm 2023-07-10T17:14:05.071310+0000 mgr.y (mgr.25390) 247 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:14:06.343 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:06 smithi192 bash[50362]: cluster 2023-07-10T17:14:05.072606+0000 mgr.y (mgr.25390) 248 : cluster 0 pgmap v169: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:06.343 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:06 smithi192 bash[50362]: cluster 2023-07-10T17:14:05.073117+0000 mgr.y (mgr.25390) 249 : cluster 0 pgmap v170: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:06.343 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:06 smithi192 bash[50362]: audit 2023-07-10T17:14:05.080156+0000 mon.a (mon.0) 585 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:06.343 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:06 smithi192 bash[50362]: audit 2023-07-10T17:14:05.169030+0000 mon.a (mon.0) 586 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:14:06.344 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:06 smithi192 bash[50362]: audit 2023-07-10T17:14:05.173892+0000 mon.a (mon.0) 587 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:06.344 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:06 smithi192 bash[50362]: audit 2023-07-10T17:14:05.176525+0000 mon.a (mon.0) 588 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:06.345 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:06 smithi192 bash[50362]: audit 2023-07-10T17:14:05.178532+0000 mon.a (mon.0) 589 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:06.345 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:06 smithi192 bash[50362]: audit 2023-07-10T17:14:05.181076+0000 mon.a (mon.0) 590 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-07-10T17:14:06.345 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:06 smithi192 bash[50362]: audit 2023-07-10T17:14:05.181378+0000 mgr.y (mgr.25390) 250 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-07-10T17:14:06.345 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:06 smithi192 bash[50362]: cephadm 2023-07-10T17:14:05.182965+0000 mgr.y (mgr.25390) 251 : cephadm 1 Upgrade: osd.5 is safe to restart 2023-07-10T17:14:06.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[65980]: cluster 2023-07-10T17:14:04.543041+0000 mgr.y (mgr.25390) 246 : cluster 0 pgmap v168: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:06.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[65980]: audit 2023-07-10T17:14:05.054691+0000 mon.a (mon.0) 581 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:06.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[65980]: audit 2023-07-10T17:14:05.065982+0000 mon.a (mon.0) 582 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:06.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[65980]: audit 2023-07-10T17:14:05.068042+0000 mon.a (mon.0) 583 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:14:06.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[65980]: audit 2023-07-10T17:14:05.069905+0000 mon.a (mon.0) 584 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:14:06.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[65980]: cephadm 2023-07-10T17:14:05.071310+0000 mgr.y (mgr.25390) 247 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:14:06.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[65980]: cluster 2023-07-10T17:14:05.072606+0000 mgr.y (mgr.25390) 248 : cluster 0 pgmap v169: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:06.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[65980]: cluster 2023-07-10T17:14:05.073117+0000 mgr.y (mgr.25390) 249 : cluster 0 pgmap v170: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:06.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[65980]: audit 2023-07-10T17:14:05.080156+0000 mon.a (mon.0) 585 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:06.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[65980]: audit 2023-07-10T17:14:05.169030+0000 mon.a (mon.0) 586 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:14:06.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[65980]: audit 2023-07-10T17:14:05.173892+0000 mon.a (mon.0) 587 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:06.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[65980]: audit 2023-07-10T17:14:05.176525+0000 mon.a (mon.0) 588 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:06.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[65980]: audit 2023-07-10T17:14:05.178532+0000 mon.a (mon.0) 589 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:06.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[65980]: audit 2023-07-10T17:14:05.181076+0000 mon.a (mon.0) 590 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-07-10T17:14:06.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[65980]: audit 2023-07-10T17:14:05.181378+0000 mgr.y (mgr.25390) 250 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-07-10T17:14:06.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[65980]: cephadm 2023-07-10T17:14:05.182965+0000 mgr.y (mgr.25390) 251 : cephadm 1 Upgrade: osd.5 is safe to restart 2023-07-10T17:14:06.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[63900]: cluster 2023-07-10T17:14:04.543041+0000 mgr.y (mgr.25390) 246 : cluster 0 pgmap v168: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:06.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[63900]: audit 2023-07-10T17:14:05.054691+0000 mon.a (mon.0) 581 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:06.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[63900]: audit 2023-07-10T17:14:05.065982+0000 mon.a (mon.0) 582 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:06.400 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[63900]: audit 2023-07-10T17:14:05.068042+0000 mon.a (mon.0) 583 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:14:06.400 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[63900]: audit 2023-07-10T17:14:05.069905+0000 mon.a (mon.0) 584 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:14:06.400 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[63900]: cephadm 2023-07-10T17:14:05.071310+0000 mgr.y (mgr.25390) 247 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:14:06.400 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[63900]: cluster 2023-07-10T17:14:05.072606+0000 mgr.y (mgr.25390) 248 : cluster 0 pgmap v169: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:06.400 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[63900]: cluster 2023-07-10T17:14:05.073117+0000 mgr.y (mgr.25390) 249 : cluster 0 pgmap v170: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:06.400 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[63900]: audit 2023-07-10T17:14:05.080156+0000 mon.a (mon.0) 585 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:06.400 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[63900]: audit 2023-07-10T17:14:05.169030+0000 mon.a (mon.0) 586 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:14:06.401 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[63900]: audit 2023-07-10T17:14:05.173892+0000 mon.a (mon.0) 587 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:06.401 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[63900]: audit 2023-07-10T17:14:05.176525+0000 mon.a (mon.0) 588 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:06.401 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[63900]: audit 2023-07-10T17:14:05.178532+0000 mon.a (mon.0) 589 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:06.401 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[63900]: audit 2023-07-10T17:14:05.181076+0000 mon.a (mon.0) 590 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-07-10T17:14:06.401 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[63900]: audit 2023-07-10T17:14:05.181378+0000 mgr.y (mgr.25390) 250 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-07-10T17:14:06.401 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:06 smithi116 bash[63900]: cephadm 2023-07-10T17:14:05.182965+0000 mgr.y (mgr.25390) 251 : cephadm 1 Upgrade: osd.5 is safe to restart 2023-07-10T17:14:08.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:08 smithi192 bash[50362]: cluster 2023-07-10T17:14:07.074320+0000 mgr.y (mgr.25390) 252 : cluster 0 pgmap v171: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:08.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:08 smithi192 bash[50362]: audit 2023-07-10T17:14:07.313491+0000 mon.a (mon.0) 591 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:08.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:08 smithi192 bash[50362]: audit 2023-07-10T17:14:07.315588+0000 mon.a (mon.0) 592 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-10T17:14:08.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:08 smithi192 bash[50362]: audit 2023-07-10T17:14:07.317056+0000 mon.a (mon.0) 593 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:14:08.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:08 smithi116 bash[65980]: cluster 2023-07-10T17:14:07.074320+0000 mgr.y (mgr.25390) 252 : cluster 0 pgmap v171: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:08.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:08 smithi116 bash[65980]: audit 2023-07-10T17:14:07.313491+0000 mon.a (mon.0) 591 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:08.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:08 smithi116 bash[65980]: audit 2023-07-10T17:14:07.315588+0000 mon.a (mon.0) 592 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-10T17:14:08.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:08 smithi116 bash[65980]: audit 2023-07-10T17:14:07.317056+0000 mon.a (mon.0) 593 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:14:08.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:08 smithi116 bash[63900]: cluster 2023-07-10T17:14:07.074320+0000 mgr.y (mgr.25390) 252 : cluster 0 pgmap v171: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:08.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:08 smithi116 bash[63900]: audit 2023-07-10T17:14:07.313491+0000 mon.a (mon.0) 591 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:08.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:08 smithi116 bash[63900]: audit 2023-07-10T17:14:07.315588+0000 mon.a (mon.0) 592 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-10T17:14:08.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:08 smithi116 bash[63900]: audit 2023-07-10T17:14:07.317056+0000 mon.a (mon.0) 593 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:14:09.451 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:09 smithi192 bash[50362]: cephadm 2023-07-10T17:14:07.302860+0000 mgr.y (mgr.25390) 253 : cephadm 1 Upgrade: Updating osd.5 2023-07-10T17:14:09.451 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:09 smithi192 bash[50362]: cephadm 2023-07-10T17:14:07.322877+0000 mgr.y (mgr.25390) 254 : cephadm 1 Deploying daemon osd.5 on smithi192 2023-07-10T17:14:09.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:09 smithi116 bash[65980]: cephadm 2023-07-10T17:14:07.302860+0000 mgr.y (mgr.25390) 253 : cephadm 1 Upgrade: Updating osd.5 2023-07-10T17:14:09.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:09 smithi116 bash[65980]: cephadm 2023-07-10T17:14:07.322877+0000 mgr.y (mgr.25390) 254 : cephadm 1 Deploying daemon osd.5 on smithi192 2023-07-10T17:14:09.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:09 smithi116 bash[63900]: cephadm 2023-07-10T17:14:07.302860+0000 mgr.y (mgr.25390) 253 : cephadm 1 Upgrade: Updating osd.5 2023-07-10T17:14:09.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:09 smithi116 bash[63900]: cephadm 2023-07-10T17:14:07.322877+0000 mgr.y (mgr.25390) 254 : cephadm 1 Deploying daemon osd.5 on smithi192 2023-07-10T17:14:10.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:10 smithi192 bash[50362]: cluster 2023-07-10T17:14:09.075499+0000 mgr.y (mgr.25390) 255 : cluster 0 pgmap v172: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:10.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:10 smithi116 bash[63900]: cluster 2023-07-10T17:14:09.075499+0000 mgr.y (mgr.25390) 255 : cluster 0 pgmap v172: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:10.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:10 smithi116 bash[65980]: cluster 2023-07-10T17:14:09.075499+0000 mgr.y (mgr.25390) 255 : cluster 0 pgmap v172: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:11.736 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:11 smithi192 systemd[1]: Stopping Ceph osd.5 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:14:12.137 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:11 smithi192 bash[29367]: debug 2023-07-10T17:14:11.729+0000 7fe6b28b8700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:14:12.137 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:11 smithi192 bash[29367]: debug 2023-07-10T17:14:11.729+0000 7fe6b28b8700 -1 osd.5 102 *** Got signal Terminated *** 2023-07-10T17:14:12.138 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:11 smithi192 bash[29367]: debug 2023-07-10T17:14:11.729+0000 7fe6b28b8700 -1 osd.5 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-10T17:14:12.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:11.076676+0000 mgr.y (mgr.25390) 256 : cluster 0 pgmap v173: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:12.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: audit 2023-07-10T17:14:11.503278+0000 mon.a (mon.0) 594 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:12.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:11.992347+0000 mon.a (mon.0) 595 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-10T17:14:12.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:11.992451+0000 mon.a (mon.0) 596 : cluster 1 osd.5 failed (root=default,host=smithi192) (connection refused reported by osd.3) 2023-07-10T17:14:12.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:11.993026+0000 mon.a (mon.0) 597 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-10T17:14:12.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:11.993532+0000 mon.a (mon.0) 598 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:12.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:11.993888+0000 mon.a (mon.0) 599 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:12.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:11.994147+0000 mon.a (mon.0) 600 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:12.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:11.994575+0000 mon.a (mon.0) 601 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-10T17:14:12.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:11.994900+0000 mon.a (mon.0) 602 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:12.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:11.995129+0000 mon.a (mon.0) 603 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:12.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:11.995478+0000 mon.a (mon.0) 604 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:12.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:11.995879+0000 mon.a (mon.0) 605 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-10T17:14:12.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:11.996281+0000 mon.a (mon.0) 606 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-10T17:14:12.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:11.996681+0000 mon.a (mon.0) 607 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-10T17:14:12.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:11.997078+0000 mon.a (mon.0) 608 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-10T17:14:12.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:11.997486+0000 mon.a (mon.0) 609 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-10T17:14:12.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.085860+0000 mon.a (mon.0) 610 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:12.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.097502+0000 mon.a (mon.0) 611 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:12.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.098593+0000 mon.a (mon.0) 612 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:12.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:11.076676+0000 mgr.y (mgr.25390) 256 : cluster 0 pgmap v173: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:12.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: audit 2023-07-10T17:14:11.503278+0000 mon.a (mon.0) 594 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:12.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:11.992347+0000 mon.a (mon.0) 595 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-10T17:14:12.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:11.992451+0000 mon.a (mon.0) 596 : cluster 1 osd.5 failed (root=default,host=smithi192) (connection refused reported by osd.3) 2023-07-10T17:14:12.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:11.993026+0000 mon.a (mon.0) 597 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-10T17:14:12.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:11.993532+0000 mon.a (mon.0) 598 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:12.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:11.993888+0000 mon.a (mon.0) 599 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:12.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:11.994147+0000 mon.a (mon.0) 600 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:12.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:11.994575+0000 mon.a (mon.0) 601 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-10T17:14:12.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:11.994900+0000 mon.a (mon.0) 602 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:12.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:11.995129+0000 mon.a (mon.0) 603 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:12.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:11.995478+0000 mon.a (mon.0) 604 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:12.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:11.995879+0000 mon.a (mon.0) 605 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-10T17:14:12.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:11.996281+0000 mon.a (mon.0) 606 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-10T17:14:12.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:11.996681+0000 mon.a (mon.0) 607 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-10T17:14:12.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:11.997078+0000 mon.a (mon.0) 608 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-10T17:14:12.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:11.997486+0000 mon.a (mon.0) 609 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-10T17:14:12.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.085860+0000 mon.a (mon.0) 610 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:12.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.097502+0000 mon.a (mon.0) 611 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:12.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.098593+0000 mon.a (mon.0) 612 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:12.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:11.076676+0000 mgr.y (mgr.25390) 256 : cluster 0 pgmap v173: 161 pgs: 161 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:12.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: audit 2023-07-10T17:14:11.503278+0000 mon.a (mon.0) 594 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:12.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:11.992347+0000 mon.a (mon.0) 595 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-10T17:14:12.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:11.992451+0000 mon.a (mon.0) 596 : cluster 1 osd.5 failed (root=default,host=smithi192) (connection refused reported by osd.3) 2023-07-10T17:14:12.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:11.993026+0000 mon.a (mon.0) 597 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-10T17:14:12.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:11.993532+0000 mon.a (mon.0) 598 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:12.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:11.993888+0000 mon.a (mon.0) 599 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:12.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:11.994147+0000 mon.a (mon.0) 600 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:12.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:11.994575+0000 mon.a (mon.0) 601 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-10T17:14:12.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:11.994900+0000 mon.a (mon.0) 602 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:12.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:11.995129+0000 mon.a (mon.0) 603 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:12.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:11.995478+0000 mon.a (mon.0) 604 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:12.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:11.995879+0000 mon.a (mon.0) 605 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-10T17:14:12.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:11.996281+0000 mon.a (mon.0) 606 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-10T17:14:12.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:11.996681+0000 mon.a (mon.0) 607 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-10T17:14:12.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:11.997078+0000 mon.a (mon.0) 608 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-10T17:14:12.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:11.997486+0000 mon.a (mon.0) 609 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-10T17:14:12.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.085860+0000 mon.a (mon.0) 610 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:12.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.097502+0000 mon.a (mon.0) 611 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:12.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:12 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.098593+0000 mon.a (mon.0) 612 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:12.887 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:12 smithi192 bash[54630]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-osd.5 2023-07-10T17:14:13.609 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.289089+0000 mon.a (mon.0) 613 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:13.609 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.300583+0000 mon.a (mon.0) 614 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:13.609 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.301016+0000 mon.a (mon.0) 615 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:13.609 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.396471+0000 mon.a (mon.0) 616 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-10T17:14:13.609 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.396744+0000 mon.a (mon.0) 617 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-10T17:14:13.610 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.397217+0000 mon.a (mon.0) 618 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-10T17:14:13.610 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.397493+0000 mon.a (mon.0) 619 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:13.610 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.397705+0000 mon.a (mon.0) 620 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-10T17:14:13.610 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.398015+0000 mon.a (mon.0) 621 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:13.610 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.398233+0000 mon.a (mon.0) 622 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:13.610 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.398436+0000 mon.a (mon.0) 623 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-10T17:14:13.610 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.398728+0000 mon.a (mon.0) 624 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:13.611 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.399059+0000 mon.a (mon.0) 625 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:13.611 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.399508+0000 mon.a (mon.0) 626 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-10T17:14:13.611 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.399635+0000 mon.a (mon.0) 627 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:13.611 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.399952+0000 mon.a (mon.0) 628 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-10T17:14:13.611 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.400121+0000 mon.a (mon.0) 629 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-10T17:14:13.611 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.499534+0000 mon.a (mon.0) 630 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:14:13.611 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.510454+0000 osd.6 (osd.6) 38 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.611 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.514286+0000 mon.a (mon.0) 631 : cluster 0 osdmap e103: 8 total, 7 up, 8 in 2023-07-10T17:14:13.612 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.516483+0000 osd.7 (osd.7) 37 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.612 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.517822+0000 osd.6 (osd.6) 39 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.612 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.519365+0000 osd.7 (osd.7) 38 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.612 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.523667+0000 osd.6 (osd.6) 40 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.612 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.529488+0000 osd.6 (osd.6) 41 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.612 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:13 smithi192 bash[50362]: cluster 2023-07-10T17:14:12.531719+0000 osd.6 (osd.6) 42 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.289089+0000 mon.a (mon.0) 613 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:13.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.300583+0000 mon.a (mon.0) 614 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:13.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.301016+0000 mon.a (mon.0) 615 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:13.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.396471+0000 mon.a (mon.0) 616 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-10T17:14:13.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.396744+0000 mon.a (mon.0) 617 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-10T17:14:13.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.397217+0000 mon.a (mon.0) 618 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-10T17:14:13.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.397493+0000 mon.a (mon.0) 619 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:13.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.397705+0000 mon.a (mon.0) 620 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-10T17:14:13.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.398015+0000 mon.a (mon.0) 621 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:13.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.398233+0000 mon.a (mon.0) 622 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:13.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.398436+0000 mon.a (mon.0) 623 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-10T17:14:13.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.398728+0000 mon.a (mon.0) 624 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:13.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.399059+0000 mon.a (mon.0) 625 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:13.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.399508+0000 mon.a (mon.0) 626 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-10T17:14:13.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.399635+0000 mon.a (mon.0) 627 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:13.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.399952+0000 mon.a (mon.0) 628 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-10T17:14:13.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.400121+0000 mon.a (mon.0) 629 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-10T17:14:13.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.499534+0000 mon.a (mon.0) 630 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:14:13.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.510454+0000 osd.6 (osd.6) 38 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.514286+0000 mon.a (mon.0) 631 : cluster 0 osdmap e103: 8 total, 7 up, 8 in 2023-07-10T17:14:13.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.516483+0000 osd.7 (osd.7) 37 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.517822+0000 osd.6 (osd.6) 39 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.519365+0000 osd.7 (osd.7) 38 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.523667+0000 osd.6 (osd.6) 40 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.529488+0000 osd.6 (osd.6) 41 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[63900]: cluster 2023-07-10T17:14:12.531719+0000 osd.6 (osd.6) 42 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.289089+0000 mon.a (mon.0) 613 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:13.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.300583+0000 mon.a (mon.0) 614 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:13.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.301016+0000 mon.a (mon.0) 615 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:13.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.396471+0000 mon.a (mon.0) 616 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-10T17:14:13.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.396744+0000 mon.a (mon.0) 617 : cluster 0 osd.5 reported immediately failed by osd.3 2023-07-10T17:14:13.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.397217+0000 mon.a (mon.0) 618 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-10T17:14:13.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.397493+0000 mon.a (mon.0) 619 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:13.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.397705+0000 mon.a (mon.0) 620 : cluster 0 osd.5 reported immediately failed by osd.2 2023-07-10T17:14:13.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.398015+0000 mon.a (mon.0) 621 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:13.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.398233+0000 mon.a (mon.0) 622 : cluster 0 osd.5 reported immediately failed by osd.4 2023-07-10T17:14:13.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.398436+0000 mon.a (mon.0) 623 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-10T17:14:13.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.398728+0000 mon.a (mon.0) 624 : cluster 0 osd.5 reported immediately failed by osd.7 2023-07-10T17:14:13.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.399059+0000 mon.a (mon.0) 625 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:13.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.399508+0000 mon.a (mon.0) 626 : cluster 0 osd.5 reported immediately failed by osd.0 2023-07-10T17:14:13.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.399635+0000 mon.a (mon.0) 627 : cluster 0 osd.5 reported immediately failed by osd.1 2023-07-10T17:14:13.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.399952+0000 mon.a (mon.0) 628 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-10T17:14:13.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.400121+0000 mon.a (mon.0) 629 : cluster 0 osd.5 reported immediately failed by osd.6 2023-07-10T17:14:13.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.499534+0000 mon.a (mon.0) 630 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:14:13.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.510454+0000 osd.6 (osd.6) 38 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.514286+0000 mon.a (mon.0) 631 : cluster 0 osdmap e103: 8 total, 7 up, 8 in 2023-07-10T17:14:13.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.516483+0000 osd.7 (osd.7) 37 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.517822+0000 osd.6 (osd.6) 39 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.519365+0000 osd.7 (osd.7) 38 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.523667+0000 osd.6 (osd.6) 40 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.529488+0000 osd.6 (osd.6) 41 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:13.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:13 smithi116 bash[65980]: cluster 2023-07-10T17:14:12.531719+0000 osd.6 (osd.6) 42 : cluster 3 failed to encode map e103 with expected crc 2023-07-10T17:14:14.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:14 smithi192 bash[50362]: cluster 2023-07-10T17:14:13.077417+0000 mgr.y (mgr.25390) 257 : cluster 0 pgmap v175: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:14.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:14 smithi192 bash[50362]: cluster 2023-07-10T17:14:13.508341+0000 osd.7 (osd.7) 39 : cluster 3 failed to encode map e104 with expected crc 2023-07-10T17:14:14.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:14 smithi192 bash[50362]: cluster 2023-07-10T17:14:13.509195+0000 osd.6 (osd.6) 43 : cluster 3 failed to encode map e104 with expected crc 2023-07-10T17:14:14.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:14 smithi192 bash[50362]: cluster 2023-07-10T17:14:13.512847+0000 mon.a (mon.0) 632 : cluster 0 osdmap e104: 8 total, 7 up, 8 in 2023-07-10T17:14:14.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:14 smithi192 bash[50362]: cluster 2023-07-10T17:14:13.514528+0000 osd.7 (osd.7) 40 : cluster 3 failed to encode map e104 with expected crc 2023-07-10T17:14:14.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:14 smithi192 bash[50362]: cluster 2023-07-10T17:14:13.515241+0000 osd.6 (osd.6) 44 : cluster 3 failed to encode map e104 with expected crc 2023-07-10T17:14:14.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:14 smithi116 bash[63900]: cluster 2023-07-10T17:14:13.077417+0000 mgr.y (mgr.25390) 257 : cluster 0 pgmap v175: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:14.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:14 smithi116 bash[63900]: cluster 2023-07-10T17:14:13.508341+0000 osd.7 (osd.7) 39 : cluster 3 failed to encode map e104 with expected crc 2023-07-10T17:14:14.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:14 smithi116 bash[63900]: cluster 2023-07-10T17:14:13.509195+0000 osd.6 (osd.6) 43 : cluster 3 failed to encode map e104 with expected crc 2023-07-10T17:14:14.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:14 smithi116 bash[63900]: cluster 2023-07-10T17:14:13.512847+0000 mon.a (mon.0) 632 : cluster 0 osdmap e104: 8 total, 7 up, 8 in 2023-07-10T17:14:14.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:14 smithi116 bash[63900]: cluster 2023-07-10T17:14:13.514528+0000 osd.7 (osd.7) 40 : cluster 3 failed to encode map e104 with expected crc 2023-07-10T17:14:14.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:14 smithi116 bash[63900]: cluster 2023-07-10T17:14:13.515241+0000 osd.6 (osd.6) 44 : cluster 3 failed to encode map e104 with expected crc 2023-07-10T17:14:14.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:14 smithi116 bash[65980]: cluster 2023-07-10T17:14:13.077417+0000 mgr.y (mgr.25390) 257 : cluster 0 pgmap v175: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:14.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:14 smithi116 bash[65980]: cluster 2023-07-10T17:14:13.508341+0000 osd.7 (osd.7) 39 : cluster 3 failed to encode map e104 with expected crc 2023-07-10T17:14:14.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:14 smithi116 bash[65980]: cluster 2023-07-10T17:14:13.509195+0000 osd.6 (osd.6) 43 : cluster 3 failed to encode map e104 with expected crc 2023-07-10T17:14:14.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:14 smithi116 bash[65980]: cluster 2023-07-10T17:14:13.512847+0000 mon.a (mon.0) 632 : cluster 0 osdmap e104: 8 total, 7 up, 8 in 2023-07-10T17:14:14.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:14 smithi116 bash[65980]: cluster 2023-07-10T17:14:13.514528+0000 osd.7 (osd.7) 40 : cluster 3 failed to encode map e104 with expected crc 2023-07-10T17:14:14.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:14 smithi116 bash[65980]: cluster 2023-07-10T17:14:13.515241+0000 osd.6 (osd.6) 44 : cluster 3 failed to encode map e104 with expected crc 2023-07-10T17:14:15.047 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:14 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.5.service: Succeeded. 2023-07-10T17:14:15.048 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:14 smithi192 systemd[1]: Stopped Ceph osd.5 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:14:15.048 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:14 smithi192 systemd[1]: Started Ceph osd.5 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:14:16.227 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:14:16.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:16 smithi192 bash[50362]: audit 2023-07-10T17:14:15.010551+0000 mon.a (mon.0) 633 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:16.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:16 smithi192 bash[50362]: audit 2023-07-10T17:14:15.020020+0000 mon.a (mon.0) 634 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:16.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:16 smithi192 bash[50362]: cluster 2023-07-10T17:14:15.078527+0000 mgr.y (mgr.25390) 258 : cluster 0 pgmap v177: 161 pgs: 3 active+undersized+degraded, 6 active+undersized, 17 stale+active+clean, 135 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 18/999 objects degraded (1.802%) 2023-07-10T17:14:16.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:16 smithi116 bash[65980]: audit 2023-07-10T17:14:15.010551+0000 mon.a (mon.0) 633 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:16.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:16 smithi116 bash[65980]: audit 2023-07-10T17:14:15.020020+0000 mon.a (mon.0) 634 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:16.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:16 smithi116 bash[65980]: cluster 2023-07-10T17:14:15.078527+0000 mgr.y (mgr.25390) 258 : cluster 0 pgmap v177: 161 pgs: 3 active+undersized+degraded, 6 active+undersized, 17 stale+active+clean, 135 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 18/999 objects degraded (1.802%) 2023-07-10T17:14:16.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:16 smithi116 bash[63900]: audit 2023-07-10T17:14:15.010551+0000 mon.a (mon.0) 633 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:16.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:16 smithi116 bash[63900]: audit 2023-07-10T17:14:15.020020+0000 mon.a (mon.0) 634 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:16.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:16 smithi116 bash[63900]: cluster 2023-07-10T17:14:15.078527+0000 mgr.y (mgr.25390) 258 : cluster 0 pgmap v177: 161 pgs: 3 active+undersized+degraded, 6 active+undersized, 17 stale+active+clean, 135 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 18/999 objects degraded (1.802%) 2023-07-10T17:14:16.967 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:14:16.967 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (7m) 51s ago 13m 10.9M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:14:16.967 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (6m) 22s ago 13m 26.0M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:14:16.967 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 error 51s ago 11m - - 2023-07-10T17:14:16.968 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (5m) 22s ago 18m 429M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:14:16.968 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (6m) 51s ago 19m 475M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:14:16.968 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (4m) 51s ago 19m 59.6M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:14:16.968 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (3m) 22s ago 18m 50.9M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 74a082a2e795 2023-07-10T17:14:16.969 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (4m) 51s ago 18m 49.9M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:14:16.969 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (7m) 51s ago 13m 9371k - 0.18.1 e5a616e4b9cf 3bd058d91353 2023-07-10T17:14:16.970 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (7m) 22s ago 13m 3808k - 0.18.1 e5a616e4b9cf 63551615c679 2023-07-10T17:14:16.970 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (2m) 51s ago 17m 53.2M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 0562f587fbca 2023-07-10T17:14:16.970 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (116s) 51s ago 17m 51.6M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 5b2a12a0bfc9 2023-07-10T17:14:16.970 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (85s) 51s ago 16m 47.9M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 88096bc0faa9 2023-07-10T17:14:16.971 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (56s) 51s ago 16m - 4096M 18.1.2-30-g8507faa7 17e8f0d79247 61ae280c471b 2023-07-10T17:14:16.971 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (27s) 22s ago 16m - 4096M 18.1.2-30-g8507faa7 17e8f0d79247 44d809c18118 2023-07-10T17:14:16.971 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 starting - - - 4096M 2023-07-10T17:14:16.971 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (15m) 22s ago 15m 61.4M 4096M 16.2.0 24ecd6d5f14c 91d6ebbcd699 2023-07-10T17:14:16.971 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (14m) 22s ago 14m 64.5M 4096M 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:14:16.971 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 error 22s ago 13m - - 2023-07-10T17:14:16.972 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (11m) 51s ago 11m 55.9M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:14:16.972 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (11m) 22s ago 11m 55.8M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:14:16.972 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (11m) 51s ago 11m 55.5M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:14:16.972 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (11m) 22s ago 11m 57.3M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:14:17.015 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:16 smithi192 bash[54906]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-07-10T17:14:17.015 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:16 smithi192 bash[54906]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2023-07-10T17:14:17.015 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:16 smithi192 bash[54906]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2023-07-10T17:14:17.015 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:16 smithi192 bash[54906]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2023-07-10T17:14:17.015 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:16 smithi192 bash[54906]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2023-07-10T17:14:17.015 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:16 smithi192 bash[54906]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-07-10T17:14:17.016 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:16 smithi192 bash[54906]: --> ceph-volume raw activate successful for osd ID: 5 2023-07-10T17:14:17.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:17 smithi192 bash[50362]: cluster 2023-07-10T17:14:16.015807+0000 mon.a (mon.0) 635 : cluster 3 Health check failed: Degraded data redundancy: 18/999 objects degraded (1.802%), 3 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:17.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:17 smithi192 bash[50362]: audit 2023-07-10T17:14:16.208847+0000 mgr.y (mgr.25390) 259 : audit 0 from='client.34623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:17.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:17 smithi116 bash[63900]: cluster 2023-07-10T17:14:16.015807+0000 mon.a (mon.0) 635 : cluster 3 Health check failed: Degraded data redundancy: 18/999 objects degraded (1.802%), 3 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:17.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:17 smithi116 bash[63900]: audit 2023-07-10T17:14:16.208847+0000 mgr.y (mgr.25390) 259 : audit 0 from='client.34623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:17.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:17 smithi116 bash[65980]: cluster 2023-07-10T17:14:16.015807+0000 mon.a (mon.0) 635 : cluster 3 Health check failed: Degraded data redundancy: 18/999 objects degraded (1.802%), 3 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:17.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:17 smithi116 bash[65980]: audit 2023-07-10T17:14:16.208847+0000 mgr.y (mgr.25390) 259 : audit 0 from='client.34623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:17.401 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:14:17.402 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:14:17.402 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:14:17.402 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:14:17.402 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:14:17.402 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:14:17.402 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:14:17.403 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:14:17.403 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2, 2023-07-10T17:14:17.403 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 5 2023-07-10T17:14:17.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:14:17.403 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:14:17.403 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:14:17.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:14:17.404 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:14:17.404 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 6, 2023-07-10T17:14:17.404 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 10 2023-07-10T17:14:17.404 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:14:17.404 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:14:17.779 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:14:17.779 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:14:17.779 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:14:17.780 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:14:17.780 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:14:17.780 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2023-07-10T17:14:17.781 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2023-07-10T17:14:17.781 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:14:17.781 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "10/23 daemons upgraded", 2023-07-10T17:14:17.782 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading osd daemons", 2023-07-10T17:14:17.782 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:14:17.782 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:14:18.214 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 2 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 78/999 objects degraded (7.808%), 13 pgs degraded 2023-07-10T17:14:18.214 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:14:18.214 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:14:18.215 INFO:teuthology.orchestra.run.smithi116.stdout: daemon prometheus.a on smithi192 is in error state 2023-07-10T17:14:18.215 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] OSD_DOWN: 1 osds down 2023-07-10T17:14:18.215 INFO:teuthology.orchestra.run.smithi116.stdout: osd.5 (root=default,host=smithi192) is down 2023-07-10T17:14:18.215 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 78/999 objects degraded (7.808%), 13 pgs degraded 2023-07-10T17:14:18.215 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.2 is active+undersized+degraded, acting [1,6] 2023-07-10T17:14:18.215 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2023-07-10T17:14:18.215 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2023-07-10T17:14:18.216 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2023-07-10T17:14:18.216 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2023-07-10T17:14:18.216 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2023-07-10T17:14:18.216 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2023-07-10T17:14:18.216 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2023-07-10T17:14:18.216 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2023-07-10T17:14:18.216 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2023-07-10T17:14:18.216 INFO:teuthology.orchestra.run.smithi116.stdout: pg 4.3 is active+undersized+degraded, acting [0,7] 2023-07-10T17:14:18.217 INFO:teuthology.orchestra.run.smithi116.stdout: pg 6.c is active+undersized+degraded, acting [3,6] 2023-07-10T17:14:18.217 INFO:teuthology.orchestra.run.smithi116.stdout: pg 6.1a is active+undersized+degraded, acting [4,1] 2023-07-10T17:14:18.350 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:18 smithi192 bash[50362]: audit 2023-07-10T17:14:16.574611+0000 mgr.y (mgr.25390) 260 : audit 0 from='client.44485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:18.350 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:18 smithi192 bash[50362]: audit 2023-07-10T17:14:16.940018+0000 mgr.y (mgr.25390) 261 : audit 0 from='client.34635 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:18.350 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:18 smithi192 bash[50362]: cluster 2023-07-10T17:14:17.079326+0000 mgr.y (mgr.25390) 262 : cluster 0 pgmap v178: 161 pgs: 13 active+undersized+degraded, 30 active+undersized, 5 stale+active+clean, 113 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 78/999 objects degraded (7.808%) 2023-07-10T17:14:18.350 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:18 smithi192 bash[50362]: audit 2023-07-10T17:14:17.396955+0000 mon.a (mon.0) 636 : audit 0 from='client.34641 172.21.15.116:0/4263778279' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:18.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:18 smithi116 bash[63900]: audit 2023-07-10T17:14:16.574611+0000 mgr.y (mgr.25390) 260 : audit 0 from='client.44485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:18.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:18 smithi116 bash[63900]: audit 2023-07-10T17:14:16.940018+0000 mgr.y (mgr.25390) 261 : audit 0 from='client.34635 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:18.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:18 smithi116 bash[63900]: cluster 2023-07-10T17:14:17.079326+0000 mgr.y (mgr.25390) 262 : cluster 0 pgmap v178: 161 pgs: 13 active+undersized+degraded, 30 active+undersized, 5 stale+active+clean, 113 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 78/999 objects degraded (7.808%) 2023-07-10T17:14:18.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:18 smithi116 bash[63900]: audit 2023-07-10T17:14:17.396955+0000 mon.a (mon.0) 636 : audit 0 from='client.34641 172.21.15.116:0/4263778279' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:18.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:18 smithi116 bash[65980]: audit 2023-07-10T17:14:16.574611+0000 mgr.y (mgr.25390) 260 : audit 0 from='client.44485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:18.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:18 smithi116 bash[65980]: audit 2023-07-10T17:14:16.940018+0000 mgr.y (mgr.25390) 261 : audit 0 from='client.34635 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:18.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:18 smithi116 bash[65980]: cluster 2023-07-10T17:14:17.079326+0000 mgr.y (mgr.25390) 262 : cluster 0 pgmap v178: 161 pgs: 13 active+undersized+degraded, 30 active+undersized, 5 stale+active+clean, 113 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 78/999 objects degraded (7.808%) 2023-07-10T17:14:18.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:18 smithi116 bash[65980]: audit 2023-07-10T17:14:17.396955+0000 mon.a (mon.0) 636 : audit 0 from='client.34641 172.21.15.116:0/4263778279' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:18.637 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:18 smithi192 bash[55213]: debug 2023-07-10T17:14:18.441+0000 7f2f9a64d700 1 -- 172.21.15.192:0/4270661918 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555aa41384e0 con 0x555aa4193000 2023-07-10T17:14:19.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:19 smithi192 bash[50362]: audit 2023-07-10T17:14:17.773609+0000 mgr.y (mgr.25390) 263 : audit 0 from='client.44503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:19.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:19 smithi192 bash[50362]: audit 2023-07-10T17:14:18.209811+0000 mon.a (mon.0) 637 : audit 0 from='client.44509 172.21.15.116:0/2953005311' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:14:19.387 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:19 smithi192 bash[55213]: debug 2023-07-10T17:14:19.001+0000 7f2fa37d2640 -1 Falling back to public interface 2023-07-10T17:14:19.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:19 smithi116 bash[63900]: audit 2023-07-10T17:14:17.773609+0000 mgr.y (mgr.25390) 263 : audit 0 from='client.44503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:19.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:19 smithi116 bash[63900]: audit 2023-07-10T17:14:18.209811+0000 mon.a (mon.0) 637 : audit 0 from='client.44509 172.21.15.116:0/2953005311' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:14:19.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:19 smithi116 bash[65980]: audit 2023-07-10T17:14:17.773609+0000 mgr.y (mgr.25390) 263 : audit 0 from='client.44503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:19.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:19 smithi116 bash[65980]: audit 2023-07-10T17:14:18.209811+0000 mon.a (mon.0) 637 : audit 0 from='client.44509 172.21.15.116:0/2953005311' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:14:20.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:20 smithi192 bash[50362]: cluster 2023-07-10T17:14:19.080578+0000 mgr.y (mgr.25390) 264 : cluster 0 pgmap v179: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2023-07-10T17:14:20.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:20 smithi116 bash[63900]: cluster 2023-07-10T17:14:19.080578+0000 mgr.y (mgr.25390) 264 : cluster 0 pgmap v179: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2023-07-10T17:14:20.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:20 smithi116 bash[65980]: cluster 2023-07-10T17:14:19.080578+0000 mgr.y (mgr.25390) 264 : cluster 0 pgmap v179: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2023-07-10T17:14:22.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:22 smithi192 bash[50362]: cluster 2023-07-10T17:14:21.082085+0000 mgr.y (mgr.25390) 265 : cluster 0 pgmap v180: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2023-07-10T17:14:22.387 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:22 smithi192 bash[55213]: debug 2023-07-10T17:14:22.129+0000 7f2fa37d2640 -1 osd.5 102 log_to_monitors true 2023-07-10T17:14:22.387 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:14:22 smithi192 bash[55213]: debug 2023-07-10T17:14:22.253+0000 7f2f91c39700 -1 osd.5 102 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-10T17:14:22.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:22 smithi116 bash[65980]: cluster 2023-07-10T17:14:21.082085+0000 mgr.y (mgr.25390) 265 : cluster 0 pgmap v180: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2023-07-10T17:14:22.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:22 smithi116 bash[63900]: cluster 2023-07-10T17:14:21.082085+0000 mgr.y (mgr.25390) 265 : cluster 0 pgmap v180: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 604 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2023-07-10T17:14:23.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:23 smithi192 bash[50362]: audit 2023-07-10T17:14:22.137592+0000 mon.a (mon.0) 638 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-10T17:14:23.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:23 smithi192 bash[50362]: audit 2023-07-10T17:14:22.137692+0000 mon.b (mon.2) 3 : audit 1 from='osd.5 [v2:172.21.15.192:6808/1055715837,v1:172.21.15.192:6809/1055715837]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-10T17:14:23.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:23 smithi192 bash[50362]: cluster 2023-07-10T17:14:22.200306+0000 osd.6 (osd.6) 45 : cluster 3 failed to encode map e105 with expected crc 2023-07-10T17:14:23.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:23 smithi192 bash[50362]: audit 2023-07-10T17:14:22.204419+0000 mon.a (mon.0) 639 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-10T17:14:23.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:23 smithi192 bash[50362]: cluster 2023-07-10T17:14:22.204686+0000 mon.a (mon.0) 640 : cluster 0 osdmap e105: 8 total, 7 up, 8 in 2023-07-10T17:14:23.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:23 smithi192 bash[50362]: audit 2023-07-10T17:14:22.207104+0000 mon.b (mon.2) 4 : audit 1 from='osd.5 [v2:172.21.15.192:6808/1055715837,v1:172.21.15.192:6809/1055715837]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:14:23.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:23 smithi192 bash[50362]: audit 2023-07-10T17:14:22.208979+0000 mon.a (mon.0) 641 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:14:23.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:23 smithi192 bash[50362]: audit 2023-07-10T17:14:22.503240+0000 mon.a (mon.0) 642 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:23.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:23 smithi192 bash[50362]: audit 2023-07-10T17:14:22.514267+0000 mon.a (mon.0) 643 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:23.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:23 smithi192 bash[50362]: cluster 2023-07-10T17:14:22.592629+0000 mon.a (mon.0) 644 : cluster 3 Health check update: Degraded data redundancy: 115/999 objects degraded (11.512%), 20 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:23.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:23 smithi192 bash[50362]: cluster 2023-07-10T17:14:22.663436+0000 osd.7 (osd.7) 41 : cluster 3 failed to encode map e105 with expected crc 2023-07-10T17:14:23.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[65980]: audit 2023-07-10T17:14:22.137592+0000 mon.a (mon.0) 638 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-10T17:14:23.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[65980]: audit 2023-07-10T17:14:22.137692+0000 mon.b (mon.2) 3 : audit 1 from='osd.5 [v2:172.21.15.192:6808/1055715837,v1:172.21.15.192:6809/1055715837]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-10T17:14:23.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[65980]: cluster 2023-07-10T17:14:22.200306+0000 osd.6 (osd.6) 45 : cluster 3 failed to encode map e105 with expected crc 2023-07-10T17:14:23.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[65980]: audit 2023-07-10T17:14:22.204419+0000 mon.a (mon.0) 639 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-10T17:14:23.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[65980]: cluster 2023-07-10T17:14:22.204686+0000 mon.a (mon.0) 640 : cluster 0 osdmap e105: 8 total, 7 up, 8 in 2023-07-10T17:14:23.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[65980]: audit 2023-07-10T17:14:22.207104+0000 mon.b (mon.2) 4 : audit 1 from='osd.5 [v2:172.21.15.192:6808/1055715837,v1:172.21.15.192:6809/1055715837]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:14:23.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[65980]: audit 2023-07-10T17:14:22.208979+0000 mon.a (mon.0) 641 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:14:23.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[65980]: audit 2023-07-10T17:14:22.503240+0000 mon.a (mon.0) 642 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:23.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[65980]: audit 2023-07-10T17:14:22.514267+0000 mon.a (mon.0) 643 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:23.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[65980]: cluster 2023-07-10T17:14:22.592629+0000 mon.a (mon.0) 644 : cluster 3 Health check update: Degraded data redundancy: 115/999 objects degraded (11.512%), 20 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:23.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[65980]: cluster 2023-07-10T17:14:22.663436+0000 osd.7 (osd.7) 41 : cluster 3 failed to encode map e105 with expected crc 2023-07-10T17:14:23.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[63900]: audit 2023-07-10T17:14:22.137592+0000 mon.a (mon.0) 638 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-10T17:14:23.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[63900]: audit 2023-07-10T17:14:22.137692+0000 mon.b (mon.2) 3 : audit 1 from='osd.5 [v2:172.21.15.192:6808/1055715837,v1:172.21.15.192:6809/1055715837]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-10T17:14:23.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[63900]: cluster 2023-07-10T17:14:22.200306+0000 osd.6 (osd.6) 45 : cluster 3 failed to encode map e105 with expected crc 2023-07-10T17:14:23.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[63900]: audit 2023-07-10T17:14:22.204419+0000 mon.a (mon.0) 639 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-10T17:14:23.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[63900]: cluster 2023-07-10T17:14:22.204686+0000 mon.a (mon.0) 640 : cluster 0 osdmap e105: 8 total, 7 up, 8 in 2023-07-10T17:14:23.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[63900]: audit 2023-07-10T17:14:22.207104+0000 mon.b (mon.2) 4 : audit 1 from='osd.5 [v2:172.21.15.192:6808/1055715837,v1:172.21.15.192:6809/1055715837]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:14:23.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[63900]: audit 2023-07-10T17:14:22.208979+0000 mon.a (mon.0) 641 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:14:23.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[63900]: audit 2023-07-10T17:14:22.503240+0000 mon.a (mon.0) 642 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:23.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[63900]: audit 2023-07-10T17:14:22.514267+0000 mon.a (mon.0) 643 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:23.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[63900]: cluster 2023-07-10T17:14:22.592629+0000 mon.a (mon.0) 644 : cluster 3 Health check update: Degraded data redundancy: 115/999 objects degraded (11.512%), 20 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:23.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:23 smithi116 bash[63900]: cluster 2023-07-10T17:14:22.663436+0000 osd.7 (osd.7) 41 : cluster 3 failed to encode map e105 with expected crc 2023-07-10T17:14:24.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:24 smithi192 bash[50362]: cluster 2023-07-10T17:14:23.083490+0000 mgr.y (mgr.25390) 266 : cluster 0 pgmap v182: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2023-07-10T17:14:24.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:24 smithi192 bash[50362]: cluster 2023-07-10T17:14:23.194139+0000 mon.a (mon.0) 645 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:14:24.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:24 smithi192 bash[50362]: cluster 2023-07-10T17:14:23.204912+0000 mon.a (mon.0) 646 : cluster 1 osd.5 [v2:172.21.15.192:6808/1055715837,v1:172.21.15.192:6809/1055715837] boot 2023-07-10T17:14:24.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:24 smithi192 bash[50362]: cluster 2023-07-10T17:14:23.204982+0000 mon.a (mon.0) 647 : cluster 0 osdmap e106: 8 total, 8 up, 8 in 2023-07-10T17:14:24.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:24 smithi192 bash[50362]: audit 2023-07-10T17:14:23.205362+0000 mon.a (mon.0) 648 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T17:14:24.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:24 smithi192 bash[50362]: cluster 2023-07-10T17:14:23.210488+0000 osd.6 (osd.6) 46 : cluster 3 failed to encode map e106 with expected crc 2023-07-10T17:14:24.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:24 smithi192 bash[50362]: audit 2023-07-10T17:14:23.313528+0000 mon.a (mon.0) 649 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:24.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:24 smithi192 bash[50362]: audit 2023-07-10T17:14:23.324780+0000 mon.a (mon.0) 650 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:24.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:24 smithi116 bash[63900]: cluster 2023-07-10T17:14:23.083490+0000 mgr.y (mgr.25390) 266 : cluster 0 pgmap v182: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2023-07-10T17:14:24.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:24 smithi116 bash[63900]: cluster 2023-07-10T17:14:23.194139+0000 mon.a (mon.0) 645 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:14:24.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:24 smithi116 bash[63900]: cluster 2023-07-10T17:14:23.204912+0000 mon.a (mon.0) 646 : cluster 1 osd.5 [v2:172.21.15.192:6808/1055715837,v1:172.21.15.192:6809/1055715837] boot 2023-07-10T17:14:24.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:24 smithi116 bash[63900]: cluster 2023-07-10T17:14:23.204982+0000 mon.a (mon.0) 647 : cluster 0 osdmap e106: 8 total, 8 up, 8 in 2023-07-10T17:14:24.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:24 smithi116 bash[63900]: audit 2023-07-10T17:14:23.205362+0000 mon.a (mon.0) 648 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T17:14:24.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:24 smithi116 bash[63900]: cluster 2023-07-10T17:14:23.210488+0000 osd.6 (osd.6) 46 : cluster 3 failed to encode map e106 with expected crc 2023-07-10T17:14:24.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:24 smithi116 bash[63900]: audit 2023-07-10T17:14:23.313528+0000 mon.a (mon.0) 649 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:24.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:24 smithi116 bash[63900]: audit 2023-07-10T17:14:23.324780+0000 mon.a (mon.0) 650 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:24.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:24 smithi116 bash[65980]: cluster 2023-07-10T17:14:23.083490+0000 mgr.y (mgr.25390) 266 : cluster 0 pgmap v182: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2023-07-10T17:14:24.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:24 smithi116 bash[65980]: cluster 2023-07-10T17:14:23.194139+0000 mon.a (mon.0) 645 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:14:24.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:24 smithi116 bash[65980]: cluster 2023-07-10T17:14:23.204912+0000 mon.a (mon.0) 646 : cluster 1 osd.5 [v2:172.21.15.192:6808/1055715837,v1:172.21.15.192:6809/1055715837] boot 2023-07-10T17:14:24.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:24 smithi116 bash[65980]: cluster 2023-07-10T17:14:23.204982+0000 mon.a (mon.0) 647 : cluster 0 osdmap e106: 8 total, 8 up, 8 in 2023-07-10T17:14:24.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:24 smithi116 bash[65980]: audit 2023-07-10T17:14:23.205362+0000 mon.a (mon.0) 648 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-10T17:14:24.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:24 smithi116 bash[65980]: cluster 2023-07-10T17:14:23.210488+0000 osd.6 (osd.6) 46 : cluster 3 failed to encode map e106 with expected crc 2023-07-10T17:14:24.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:24 smithi116 bash[65980]: audit 2023-07-10T17:14:23.313528+0000 mon.a (mon.0) 649 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:24.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:24 smithi116 bash[65980]: audit 2023-07-10T17:14:23.324780+0000 mon.a (mon.0) 650 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:25.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:25 smithi192 bash[50362]: cluster 2023-07-10T17:14:24.203255+0000 osd.7 (osd.7) 42 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:25 smithi192 bash[50362]: cluster 2023-07-10T17:14:24.204177+0000 osd.6 (osd.6) 47 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:25 smithi192 bash[50362]: cluster 2023-07-10T17:14:24.206282+0000 mon.a (mon.0) 651 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2023-07-10T17:14:25.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:25 smithi192 bash[50362]: cluster 2023-07-10T17:14:24.210178+0000 osd.6 (osd.6) 48 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:25 smithi192 bash[50362]: cluster 2023-07-10T17:14:24.210303+0000 osd.7 (osd.7) 43 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:25 smithi192 bash[50362]: cluster 2023-07-10T17:14:24.213640+0000 osd.6 (osd.6) 49 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:25 smithi192 bash[50362]: cluster 2023-07-10T17:14:24.214041+0000 osd.7 (osd.7) 44 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:25 smithi192 bash[50362]: cluster 2023-07-10T17:14:24.216469+0000 osd.6 (osd.6) 50 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:25 smithi192 bash[50362]: cluster 2023-07-10T17:14:24.216491+0000 osd.7 (osd.7) 45 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:25 smithi192 bash[50362]: cluster 2023-07-10T17:14:24.218552+0000 osd.7 (osd.7) 46 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:25 smithi192 bash[50362]: cluster 2023-07-10T17:14:24.220263+0000 osd.6 (osd.6) 51 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:25 smithi192 bash[50362]: cluster 2023-07-10T17:14:24.221541+0000 osd.7 (osd.7) 47 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[65980]: cluster 2023-07-10T17:14:24.203255+0000 osd.7 (osd.7) 42 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[65980]: cluster 2023-07-10T17:14:24.204177+0000 osd.6 (osd.6) 47 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[65980]: cluster 2023-07-10T17:14:24.206282+0000 mon.a (mon.0) 651 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2023-07-10T17:14:25.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[65980]: cluster 2023-07-10T17:14:24.210178+0000 osd.6 (osd.6) 48 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[65980]: cluster 2023-07-10T17:14:24.210303+0000 osd.7 (osd.7) 43 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[65980]: cluster 2023-07-10T17:14:24.213640+0000 osd.6 (osd.6) 49 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[65980]: cluster 2023-07-10T17:14:24.214041+0000 osd.7 (osd.7) 44 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[65980]: cluster 2023-07-10T17:14:24.216469+0000 osd.6 (osd.6) 50 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[65980]: cluster 2023-07-10T17:14:24.216491+0000 osd.7 (osd.7) 45 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[65980]: cluster 2023-07-10T17:14:24.218552+0000 osd.7 (osd.7) 46 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[65980]: cluster 2023-07-10T17:14:24.220263+0000 osd.6 (osd.6) 51 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[65980]: cluster 2023-07-10T17:14:24.221541+0000 osd.7 (osd.7) 47 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[63900]: cluster 2023-07-10T17:14:24.203255+0000 osd.7 (osd.7) 42 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[63900]: cluster 2023-07-10T17:14:24.204177+0000 osd.6 (osd.6) 47 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[63900]: cluster 2023-07-10T17:14:24.206282+0000 mon.a (mon.0) 651 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2023-07-10T17:14:25.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[63900]: cluster 2023-07-10T17:14:24.210178+0000 osd.6 (osd.6) 48 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[63900]: cluster 2023-07-10T17:14:24.210303+0000 osd.7 (osd.7) 43 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[63900]: cluster 2023-07-10T17:14:24.213640+0000 osd.6 (osd.6) 49 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[63900]: cluster 2023-07-10T17:14:24.214041+0000 osd.7 (osd.7) 44 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[63900]: cluster 2023-07-10T17:14:24.216469+0000 osd.6 (osd.6) 50 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[63900]: cluster 2023-07-10T17:14:24.216491+0000 osd.7 (osd.7) 45 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[63900]: cluster 2023-07-10T17:14:24.218552+0000 osd.7 (osd.7) 46 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[63900]: cluster 2023-07-10T17:14:24.220263+0000 osd.6 (osd.6) 51 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:25.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:25 smithi116 bash[63900]: cluster 2023-07-10T17:14:24.221541+0000 osd.7 (osd.7) 47 : cluster 3 failed to encode map e107 with expected crc 2023-07-10T17:14:26.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:26 smithi192 bash[50362]: cluster 2023-07-10T17:14:25.084525+0000 mgr.y (mgr.25390) 267 : cluster 0 pgmap v185: 161 pgs: 5 peering, 19 active+undersized+degraded, 35 active+undersized, 102 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-10T17:14:26.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:26 smithi116 bash[63900]: cluster 2023-07-10T17:14:25.084525+0000 mgr.y (mgr.25390) 267 : cluster 0 pgmap v185: 161 pgs: 5 peering, 19 active+undersized+degraded, 35 active+undersized, 102 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-10T17:14:26.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:26 smithi116 bash[65980]: cluster 2023-07-10T17:14:25.084525+0000 mgr.y (mgr.25390) 267 : cluster 0 pgmap v185: 161 pgs: 5 peering, 19 active+undersized+degraded, 35 active+undersized, 102 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 114/999 objects degraded (11.411%) 2023-07-10T17:14:27.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:27 smithi192 bash[50362]: audit 2023-07-10T17:14:26.535203+0000 mon.a (mon.0) 652 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:27.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:27 smithi192 bash[50362]: cluster 2023-07-10T17:14:27.086077+0000 mgr.y (mgr.25390) 268 : cluster 0 pgmap v186: 161 pgs: 5 peering, 13 active+undersized+degraded, 19 active+undersized, 124 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 73/999 objects degraded (7.307%) 2023-07-10T17:14:27.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:27 smithi116 bash[63900]: audit 2023-07-10T17:14:26.535203+0000 mon.a (mon.0) 652 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:27.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:27 smithi116 bash[63900]: cluster 2023-07-10T17:14:27.086077+0000 mgr.y (mgr.25390) 268 : cluster 0 pgmap v186: 161 pgs: 5 peering, 13 active+undersized+degraded, 19 active+undersized, 124 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 73/999 objects degraded (7.307%) 2023-07-10T17:14:27.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:27 smithi116 bash[65980]: audit 2023-07-10T17:14:26.535203+0000 mon.a (mon.0) 652 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:27.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:27 smithi116 bash[65980]: cluster 2023-07-10T17:14:27.086077+0000 mgr.y (mgr.25390) 268 : cluster 0 pgmap v186: 161 pgs: 5 peering, 13 active+undersized+degraded, 19 active+undersized, 124 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 73/999 objects degraded (7.307%) 2023-07-10T17:14:28.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:28 smithi192 bash[50362]: cluster 2023-07-10T17:14:27.594376+0000 mon.a (mon.0) 653 : cluster 3 Health check update: Degraded data redundancy: 73/999 objects degraded (7.307%), 13 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:28.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:28 smithi116 bash[63900]: cluster 2023-07-10T17:14:27.594376+0000 mon.a (mon.0) 653 : cluster 3 Health check update: Degraded data redundancy: 73/999 objects degraded (7.307%), 13 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:28.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:28 smithi116 bash[65980]: cluster 2023-07-10T17:14:27.594376+0000 mon.a (mon.0) 653 : cluster 3 Health check update: Degraded data redundancy: 73/999 objects degraded (7.307%), 13 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:29.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:29 smithi192 bash[50362]: cluster 2023-07-10T17:14:29.087381+0000 mgr.y (mgr.25390) 269 : cluster 0 pgmap v187: 161 pgs: 5 peering, 156 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:29.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:29 smithi116 bash[65980]: cluster 2023-07-10T17:14:29.087381+0000 mgr.y (mgr.25390) 269 : cluster 0 pgmap v187: 161 pgs: 5 peering, 156 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:29.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:29 smithi116 bash[63900]: cluster 2023-07-10T17:14:29.087381+0000 mgr.y (mgr.25390) 269 : cluster 0 pgmap v187: 161 pgs: 5 peering, 156 active+clean; 604 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:30.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:30 smithi192 bash[50362]: cluster 2023-07-10T17:14:29.543991+0000 mon.a (mon.0) 654 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/999 objects degraded (7.307%), 13 pgs degraded) 2023-07-10T17:14:30.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:30 smithi116 bash[63900]: cluster 2023-07-10T17:14:29.543991+0000 mon.a (mon.0) 654 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/999 objects degraded (7.307%), 13 pgs degraded) 2023-07-10T17:14:30.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:30 smithi116 bash[65980]: cluster 2023-07-10T17:14:29.543991+0000 mon.a (mon.0) 654 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/999 objects degraded (7.307%), 13 pgs degraded) 2023-07-10T17:14:31.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:31 smithi192 bash[50362]: cluster 2023-07-10T17:14:31.088644+0000 mgr.y (mgr.25390) 270 : cluster 0 pgmap v188: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:31.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:31 smithi116 bash[63900]: cluster 2023-07-10T17:14:31.088644+0000 mgr.y (mgr.25390) 270 : cluster 0 pgmap v188: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:31.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:31 smithi116 bash[65980]: cluster 2023-07-10T17:14:31.088644+0000 mgr.y (mgr.25390) 270 : cluster 0 pgmap v188: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:33.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:33 smithi192 bash[50362]: cluster 2023-07-10T17:14:33.089911+0000 mgr.y (mgr.25390) 271 : cluster 0 pgmap v189: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:33.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:33 smithi192 bash[50362]: audit 2023-07-10T17:14:33.475941+0000 mon.a (mon.0) 655 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:33.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:33 smithi192 bash[50362]: audit 2023-07-10T17:14:33.486699+0000 mon.a (mon.0) 656 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:33.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:33 smithi192 bash[50362]: audit 2023-07-10T17:14:33.489148+0000 mon.a (mon.0) 657 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:14:33.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:33 smithi192 bash[50362]: audit 2023-07-10T17:14:33.490992+0000 mon.a (mon.0) 658 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:14:33.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:33 smithi192 bash[50362]: audit 2023-07-10T17:14:33.501809+0000 mon.a (mon.0) 659 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:33.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:33 smithi192 bash[50362]: audit 2023-07-10T17:14:33.590865+0000 mon.a (mon.0) 660 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:14:33.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:33 smithi192 bash[50362]: audit 2023-07-10T17:14:33.597101+0000 mon.a (mon.0) 661 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:33.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:33 smithi116 bash[63900]: cluster 2023-07-10T17:14:33.089911+0000 mgr.y (mgr.25390) 271 : cluster 0 pgmap v189: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:33.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:33 smithi116 bash[63900]: audit 2023-07-10T17:14:33.475941+0000 mon.a (mon.0) 655 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:33.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:33 smithi116 bash[63900]: audit 2023-07-10T17:14:33.486699+0000 mon.a (mon.0) 656 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:33.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:33 smithi116 bash[63900]: audit 2023-07-10T17:14:33.489148+0000 mon.a (mon.0) 657 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:14:33.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:33 smithi116 bash[63900]: audit 2023-07-10T17:14:33.490992+0000 mon.a (mon.0) 658 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:14:33.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:33 smithi116 bash[63900]: audit 2023-07-10T17:14:33.501809+0000 mon.a (mon.0) 659 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:33.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:33 smithi116 bash[63900]: audit 2023-07-10T17:14:33.590865+0000 mon.a (mon.0) 660 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:14:33.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:33 smithi116 bash[63900]: audit 2023-07-10T17:14:33.597101+0000 mon.a (mon.0) 661 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:33.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:33 smithi116 bash[65980]: cluster 2023-07-10T17:14:33.089911+0000 mgr.y (mgr.25390) 271 : cluster 0 pgmap v189: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:33.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:33 smithi116 bash[65980]: audit 2023-07-10T17:14:33.475941+0000 mon.a (mon.0) 655 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:33.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:33 smithi116 bash[65980]: audit 2023-07-10T17:14:33.486699+0000 mon.a (mon.0) 656 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:33.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:33 smithi116 bash[65980]: audit 2023-07-10T17:14:33.489148+0000 mon.a (mon.0) 657 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:14:33.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:33 smithi116 bash[65980]: audit 2023-07-10T17:14:33.490992+0000 mon.a (mon.0) 658 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:14:33.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:33 smithi116 bash[65980]: audit 2023-07-10T17:14:33.501809+0000 mon.a (mon.0) 659 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:33.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:33 smithi116 bash[65980]: audit 2023-07-10T17:14:33.590865+0000 mon.a (mon.0) 660 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:14:33.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:33 smithi116 bash[65980]: audit 2023-07-10T17:14:33.597101+0000 mon.a (mon.0) 661 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:34.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:34 smithi192 bash[50362]: cephadm 2023-07-10T17:14:33.492491+0000 mgr.y (mgr.25390) 272 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:14:34.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:34 smithi192 bash[50362]: cluster 2023-07-10T17:14:33.493903+0000 mgr.y (mgr.25390) 273 : cluster 0 pgmap v190: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:34.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:34 smithi192 bash[50362]: cluster 2023-07-10T17:14:33.494511+0000 mgr.y (mgr.25390) 274 : cluster 0 pgmap v191: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:34.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:34 smithi192 bash[50362]: audit 2023-07-10T17:14:33.599990+0000 mon.a (mon.0) 662 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:34.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:34 smithi192 bash[50362]: audit 2023-07-10T17:14:33.606804+0000 mon.a (mon.0) 663 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:34.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:34 smithi192 bash[50362]: audit 2023-07-10T17:14:33.616869+0000 mon.a (mon.0) 664 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-07-10T17:14:34.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:34 smithi192 bash[50362]: audit 2023-07-10T17:14:33.617200+0000 mgr.y (mgr.25390) 275 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-07-10T17:14:34.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:34 smithi192 bash[50362]: cephadm 2023-07-10T17:14:33.619034+0000 mgr.y (mgr.25390) 276 : cephadm 1 Upgrade: osd.6 is safe to restart 2023-07-10T17:14:34.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:34 smithi116 bash[63900]: cephadm 2023-07-10T17:14:33.492491+0000 mgr.y (mgr.25390) 272 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:14:34.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:34 smithi116 bash[63900]: cluster 2023-07-10T17:14:33.493903+0000 mgr.y (mgr.25390) 273 : cluster 0 pgmap v190: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:34.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:34 smithi116 bash[63900]: cluster 2023-07-10T17:14:33.494511+0000 mgr.y (mgr.25390) 274 : cluster 0 pgmap v191: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:34.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:34 smithi116 bash[63900]: audit 2023-07-10T17:14:33.599990+0000 mon.a (mon.0) 662 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:34.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:34 smithi116 bash[63900]: audit 2023-07-10T17:14:33.606804+0000 mon.a (mon.0) 663 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:34.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:34 smithi116 bash[63900]: audit 2023-07-10T17:14:33.616869+0000 mon.a (mon.0) 664 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-07-10T17:14:34.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:34 smithi116 bash[63900]: audit 2023-07-10T17:14:33.617200+0000 mgr.y (mgr.25390) 275 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-07-10T17:14:34.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:34 smithi116 bash[63900]: cephadm 2023-07-10T17:14:33.619034+0000 mgr.y (mgr.25390) 276 : cephadm 1 Upgrade: osd.6 is safe to restart 2023-07-10T17:14:34.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:34 smithi116 bash[65980]: cephadm 2023-07-10T17:14:33.492491+0000 mgr.y (mgr.25390) 272 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:14:34.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:34 smithi116 bash[65980]: cluster 2023-07-10T17:14:33.493903+0000 mgr.y (mgr.25390) 273 : cluster 0 pgmap v190: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:34.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:34 smithi116 bash[65980]: cluster 2023-07-10T17:14:33.494511+0000 mgr.y (mgr.25390) 274 : cluster 0 pgmap v191: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:34.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:34 smithi116 bash[65980]: audit 2023-07-10T17:14:33.599990+0000 mon.a (mon.0) 662 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:34.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:34 smithi116 bash[65980]: audit 2023-07-10T17:14:33.606804+0000 mon.a (mon.0) 663 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:34.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:34 smithi116 bash[65980]: audit 2023-07-10T17:14:33.616869+0000 mon.a (mon.0) 664 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-07-10T17:14:34.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:34 smithi116 bash[65980]: audit 2023-07-10T17:14:33.617200+0000 mgr.y (mgr.25390) 275 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-07-10T17:14:34.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:34 smithi116 bash[65980]: cephadm 2023-07-10T17:14:33.619034+0000 mgr.y (mgr.25390) 276 : cephadm 1 Upgrade: osd.6 is safe to restart 2023-07-10T17:14:37.077 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:36 smithi192 bash[50362]: cluster 2023-07-10T17:14:35.495493+0000 mgr.y (mgr.25390) 277 : cluster 0 pgmap v192: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:37.078 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:36 smithi192 bash[50362]: cephadm 2023-07-10T17:14:35.788749+0000 mgr.y (mgr.25390) 278 : cephadm 1 Upgrade: Updating osd.6 2023-07-10T17:14:37.078 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:36 smithi192 bash[50362]: audit 2023-07-10T17:14:35.798332+0000 mon.a (mon.0) 665 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:37.078 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:36 smithi192 bash[50362]: audit 2023-07-10T17:14:35.800089+0000 mon.a (mon.0) 666 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-10T17:14:37.078 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:36 smithi192 bash[50362]: audit 2023-07-10T17:14:35.801795+0000 mon.a (mon.0) 667 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:14:37.078 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:36 smithi192 bash[50362]: cephadm 2023-07-10T17:14:35.806328+0000 mgr.y (mgr.25390) 279 : cephadm 1 Deploying daemon osd.6 on smithi192 2023-07-10T17:14:37.078 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:36 smithi192 bash[50362]: audit 2023-07-10T17:14:36.562950+0000 mon.a (mon.0) 668 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:37.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:36 smithi116 bash[63900]: cluster 2023-07-10T17:14:35.495493+0000 mgr.y (mgr.25390) 277 : cluster 0 pgmap v192: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:37.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:36 smithi116 bash[63900]: cephadm 2023-07-10T17:14:35.788749+0000 mgr.y (mgr.25390) 278 : cephadm 1 Upgrade: Updating osd.6 2023-07-10T17:14:37.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:36 smithi116 bash[63900]: audit 2023-07-10T17:14:35.798332+0000 mon.a (mon.0) 665 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:37.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:36 smithi116 bash[63900]: audit 2023-07-10T17:14:35.800089+0000 mon.a (mon.0) 666 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-10T17:14:37.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:36 smithi116 bash[63900]: audit 2023-07-10T17:14:35.801795+0000 mon.a (mon.0) 667 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:14:37.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:36 smithi116 bash[63900]: cephadm 2023-07-10T17:14:35.806328+0000 mgr.y (mgr.25390) 279 : cephadm 1 Deploying daemon osd.6 on smithi192 2023-07-10T17:14:37.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:36 smithi116 bash[63900]: audit 2023-07-10T17:14:36.562950+0000 mon.a (mon.0) 668 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:37.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:36 smithi116 bash[65980]: cluster 2023-07-10T17:14:35.495493+0000 mgr.y (mgr.25390) 277 : cluster 0 pgmap v192: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:37.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:36 smithi116 bash[65980]: cephadm 2023-07-10T17:14:35.788749+0000 mgr.y (mgr.25390) 278 : cephadm 1 Upgrade: Updating osd.6 2023-07-10T17:14:37.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:36 smithi116 bash[65980]: audit 2023-07-10T17:14:35.798332+0000 mon.a (mon.0) 665 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:37.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:36 smithi116 bash[65980]: audit 2023-07-10T17:14:35.800089+0000 mon.a (mon.0) 666 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-10T17:14:37.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:36 smithi116 bash[65980]: audit 2023-07-10T17:14:35.801795+0000 mon.a (mon.0) 667 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:14:37.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:36 smithi116 bash[65980]: cephadm 2023-07-10T17:14:35.806328+0000 mgr.y (mgr.25390) 279 : cephadm 1 Deploying daemon osd.6 on smithi192 2023-07-10T17:14:37.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:36 smithi116 bash[65980]: audit 2023-07-10T17:14:36.562950+0000 mon.a (mon.0) 668 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:39.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:38 smithi192 bash[50362]: cluster 2023-07-10T17:14:37.497042+0000 mgr.y (mgr.25390) 280 : cluster 0 pgmap v193: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:39.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:38 smithi116 bash[63900]: cluster 2023-07-10T17:14:37.497042+0000 mgr.y (mgr.25390) 280 : cluster 0 pgmap v193: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:39.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:38 smithi116 bash[65980]: cluster 2023-07-10T17:14:37.497042+0000 mgr.y (mgr.25390) 280 : cluster 0 pgmap v193: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:40.337 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:40 smithi192 systemd[1]: Stopping Ceph osd.6 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:14:40.588 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[32145]: debug 2023-07-10T17:14:40.332+0000 7f9aa17cb700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:14:40.588 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[32145]: debug 2023-07-10T17:14:40.332+0000 7f9aa17cb700 -1 osd.6 107 *** Got signal Terminated *** 2023-07-10T17:14:40.589 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[32145]: debug 2023-07-10T17:14:40.332+0000 7f9aa17cb700 -1 osd.6 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-10T17:14:40.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[50362]: cluster 2023-07-10T17:14:39.497988+0000 mgr.y (mgr.25390) 281 : cluster 0 pgmap v194: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:40.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.596391+0000 mon.a (mon.0) 669 : cluster 0 osd.6 reported immediately failed by osd.3 2023-07-10T17:14:40.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.596505+0000 mon.a (mon.0) 670 : cluster 1 osd.6 failed (root=default,host=smithi192) (connection refused reported by osd.3) 2023-07-10T17:14:40.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.597050+0000 mon.a (mon.0) 671 : cluster 0 osd.6 reported immediately failed by osd.3 2023-07-10T17:14:40.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.597557+0000 mon.a (mon.0) 672 : cluster 0 osd.6 reported immediately failed by osd.7 2023-07-10T17:14:40.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.597814+0000 mon.a (mon.0) 673 : cluster 0 osd.6 reported immediately failed by osd.4 2023-07-10T17:14:40.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.598058+0000 mon.a (mon.0) 674 : cluster 0 osd.6 reported immediately failed by osd.0 2023-07-10T17:14:40.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.598561+0000 mon.a (mon.0) 675 : cluster 0 osd.6 reported immediately failed by osd.5 2023-07-10T17:14:40.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.598804+0000 mon.a (mon.0) 676 : cluster 0 osd.6 reported immediately failed by osd.7 2023-07-10T17:14:40.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.598939+0000 mon.a (mon.0) 677 : cluster 0 osd.6 reported immediately failed by osd.4 2023-07-10T17:14:40.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.599410+0000 mon.a (mon.0) 678 : cluster 0 osd.6 reported immediately failed by osd.2 2023-07-10T17:14:40.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.599678+0000 mon.a (mon.0) 679 : cluster 0 osd.6 reported immediately failed by osd.5 2023-07-10T17:14:40.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.600071+0000 mon.a (mon.0) 680 : cluster 0 osd.6 reported immediately failed by osd.0 2023-07-10T17:14:40.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.600363+0000 mon.a (mon.0) 681 : cluster 0 osd.6 reported immediately failed by osd.1 2023-07-10T17:14:40.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.600832+0000 mon.a (mon.0) 682 : cluster 0 osd.6 reported immediately failed by osd.1 2023-07-10T17:14:40.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:40 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.601090+0000 mon.a (mon.0) 683 : cluster 0 osd.6 reported immediately failed by osd.2 2023-07-10T17:14:41.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[63900]: cluster 2023-07-10T17:14:39.497988+0000 mgr.y (mgr.25390) 281 : cluster 0 pgmap v194: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:41.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.596391+0000 mon.a (mon.0) 669 : cluster 0 osd.6 reported immediately failed by osd.3 2023-07-10T17:14:41.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.596505+0000 mon.a (mon.0) 670 : cluster 1 osd.6 failed (root=default,host=smithi192) (connection refused reported by osd.3) 2023-07-10T17:14:41.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.597050+0000 mon.a (mon.0) 671 : cluster 0 osd.6 reported immediately failed by osd.3 2023-07-10T17:14:41.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.597557+0000 mon.a (mon.0) 672 : cluster 0 osd.6 reported immediately failed by osd.7 2023-07-10T17:14:41.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.597814+0000 mon.a (mon.0) 673 : cluster 0 osd.6 reported immediately failed by osd.4 2023-07-10T17:14:41.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.598058+0000 mon.a (mon.0) 674 : cluster 0 osd.6 reported immediately failed by osd.0 2023-07-10T17:14:41.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.598561+0000 mon.a (mon.0) 675 : cluster 0 osd.6 reported immediately failed by osd.5 2023-07-10T17:14:41.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.598804+0000 mon.a (mon.0) 676 : cluster 0 osd.6 reported immediately failed by osd.7 2023-07-10T17:14:41.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.598939+0000 mon.a (mon.0) 677 : cluster 0 osd.6 reported immediately failed by osd.4 2023-07-10T17:14:41.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.599410+0000 mon.a (mon.0) 678 : cluster 0 osd.6 reported immediately failed by osd.2 2023-07-10T17:14:41.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.599678+0000 mon.a (mon.0) 679 : cluster 0 osd.6 reported immediately failed by osd.5 2023-07-10T17:14:41.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.600071+0000 mon.a (mon.0) 680 : cluster 0 osd.6 reported immediately failed by osd.0 2023-07-10T17:14:41.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.600363+0000 mon.a (mon.0) 681 : cluster 0 osd.6 reported immediately failed by osd.1 2023-07-10T17:14:41.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.600832+0000 mon.a (mon.0) 682 : cluster 0 osd.6 reported immediately failed by osd.1 2023-07-10T17:14:41.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.601090+0000 mon.a (mon.0) 683 : cluster 0 osd.6 reported immediately failed by osd.2 2023-07-10T17:14:41.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[65980]: cluster 2023-07-10T17:14:39.497988+0000 mgr.y (mgr.25390) 281 : cluster 0 pgmap v194: 161 pgs: 161 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:41.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.596391+0000 mon.a (mon.0) 669 : cluster 0 osd.6 reported immediately failed by osd.3 2023-07-10T17:14:41.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.596505+0000 mon.a (mon.0) 670 : cluster 1 osd.6 failed (root=default,host=smithi192) (connection refused reported by osd.3) 2023-07-10T17:14:41.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.597050+0000 mon.a (mon.0) 671 : cluster 0 osd.6 reported immediately failed by osd.3 2023-07-10T17:14:41.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.597557+0000 mon.a (mon.0) 672 : cluster 0 osd.6 reported immediately failed by osd.7 2023-07-10T17:14:41.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.597814+0000 mon.a (mon.0) 673 : cluster 0 osd.6 reported immediately failed by osd.4 2023-07-10T17:14:41.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.598058+0000 mon.a (mon.0) 674 : cluster 0 osd.6 reported immediately failed by osd.0 2023-07-10T17:14:41.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.598561+0000 mon.a (mon.0) 675 : cluster 0 osd.6 reported immediately failed by osd.5 2023-07-10T17:14:41.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.598804+0000 mon.a (mon.0) 676 : cluster 0 osd.6 reported immediately failed by osd.7 2023-07-10T17:14:41.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.598939+0000 mon.a (mon.0) 677 : cluster 0 osd.6 reported immediately failed by osd.4 2023-07-10T17:14:41.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.599410+0000 mon.a (mon.0) 678 : cluster 0 osd.6 reported immediately failed by osd.2 2023-07-10T17:14:41.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.599678+0000 mon.a (mon.0) 679 : cluster 0 osd.6 reported immediately failed by osd.5 2023-07-10T17:14:41.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.600071+0000 mon.a (mon.0) 680 : cluster 0 osd.6 reported immediately failed by osd.0 2023-07-10T17:14:41.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.600363+0000 mon.a (mon.0) 681 : cluster 0 osd.6 reported immediately failed by osd.1 2023-07-10T17:14:41.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.600832+0000 mon.a (mon.0) 682 : cluster 0 osd.6 reported immediately failed by osd.1 2023-07-10T17:14:41.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:40 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.601090+0000 mon.a (mon.0) 683 : cluster 0 osd.6 reported immediately failed by osd.2 2023-07-10T17:14:41.637 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[56825]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-osd.6 2023-07-10T17:14:42.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.803471+0000 mon.a (mon.0) 684 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:14:42.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.815060+0000 mon.a (mon.0) 685 : cluster 0 osdmap e108: 8 total, 7 up, 8 in 2023-07-10T17:14:42.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.815626+0000 osd.7 (osd.7) 48 : cluster 3 failed to encode map e108 with expected crc 2023-07-10T17:14:42.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.820187+0000 osd.7 (osd.7) 49 : cluster 3 failed to encode map e108 with expected crc 2023-07-10T17:14:42.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.822085+0000 osd.7 (osd.7) 50 : cluster 3 failed to encode map e108 with expected crc 2023-07-10T17:14:42.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.824246+0000 osd.7 (osd.7) 51 : cluster 3 failed to encode map e108 with expected crc 2023-07-10T17:14:42.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: cluster 2023-07-10T17:14:40.826658+0000 osd.7 (osd.7) 52 : cluster 3 failed to encode map e108 with expected crc 2023-07-10T17:14:42.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: audit 2023-07-10T17:14:41.281701+0000 mon.a (mon.0) 686 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 6]}]: dispatch 2023-07-10T17:14:42.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: audit 2023-07-10T17:14:41.282235+0000 mon.a (mon.0) 687 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2, 3, 0]}]: dispatch 2023-07-10T17:14:42.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: audit 2023-07-10T17:14:41.282662+0000 mon.a (mon.0) 688 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 6]}]: dispatch 2023-07-10T17:14:42.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: audit 2023-07-10T17:14:41.283108+0000 mon.a (mon.0) 689 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2023-07-10T17:14:42.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: audit 2023-07-10T17:14:41.283574+0000 mon.a (mon.0) 690 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]: dispatch 2023-07-10T17:14:42.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: audit 2023-07-10T17:14:41.284000+0000 mon.a (mon.0) 691 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 6]}]: dispatch 2023-07-10T17:14:42.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: audit 2023-07-10T17:14:41.284426+0000 mon.a (mon.0) 692 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [4, 6]}]: dispatch 2023-07-10T17:14:42.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: audit 2023-07-10T17:14:41.284835+0000 mon.a (mon.0) 693 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 6]}]: dispatch 2023-07-10T17:14:42.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: audit 2023-07-10T17:14:41.285250+0000 mon.a (mon.0) 694 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 6]}]: dispatch 2023-07-10T17:14:42.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: audit 2023-07-10T17:14:41.817406+0000 mon.a (mon.0) 695 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 6]}]': finished 2023-07-10T17:14:42.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: audit 2023-07-10T17:14:41.817637+0000 mon.a (mon.0) 696 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2, 3, 0]}]': finished 2023-07-10T17:14:42.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: audit 2023-07-10T17:14:41.817831+0000 mon.a (mon.0) 697 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 6]}]': finished 2023-07-10T17:14:42.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: audit 2023-07-10T17:14:41.818016+0000 mon.a (mon.0) 698 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2023-07-10T17:14:42.141 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: audit 2023-07-10T17:14:41.818198+0000 mon.a (mon.0) 699 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]': finished 2023-07-10T17:14:42.141 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: audit 2023-07-10T17:14:41.818380+0000 mon.a (mon.0) 700 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 6]}]': finished 2023-07-10T17:14:42.141 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:41 smithi192 bash[50362]: audit 2023-07-10T17:14:41.818544+0000 mon.a (mon.0) 701 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [4, 6]}]': finished 2023-07-10T17:14:42.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.803471+0000 mon.a (mon.0) 684 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:14:42.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.815060+0000 mon.a (mon.0) 685 : cluster 0 osdmap e108: 8 total, 7 up, 8 in 2023-07-10T17:14:42.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.815626+0000 osd.7 (osd.7) 48 : cluster 3 failed to encode map e108 with expected crc 2023-07-10T17:14:42.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.820187+0000 osd.7 (osd.7) 49 : cluster 3 failed to encode map e108 with expected crc 2023-07-10T17:14:42.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.822085+0000 osd.7 (osd.7) 50 : cluster 3 failed to encode map e108 with expected crc 2023-07-10T17:14:42.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.824246+0000 osd.7 (osd.7) 51 : cluster 3 failed to encode map e108 with expected crc 2023-07-10T17:14:42.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: cluster 2023-07-10T17:14:40.826658+0000 osd.7 (osd.7) 52 : cluster 3 failed to encode map e108 with expected crc 2023-07-10T17:14:42.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: audit 2023-07-10T17:14:41.281701+0000 mon.a (mon.0) 686 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 6]}]: dispatch 2023-07-10T17:14:42.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: audit 2023-07-10T17:14:41.282235+0000 mon.a (mon.0) 687 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2, 3, 0]}]: dispatch 2023-07-10T17:14:42.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: audit 2023-07-10T17:14:41.282662+0000 mon.a (mon.0) 688 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 6]}]: dispatch 2023-07-10T17:14:42.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: audit 2023-07-10T17:14:41.283108+0000 mon.a (mon.0) 689 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2023-07-10T17:14:42.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: audit 2023-07-10T17:14:41.283574+0000 mon.a (mon.0) 690 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]: dispatch 2023-07-10T17:14:42.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: audit 2023-07-10T17:14:41.284000+0000 mon.a (mon.0) 691 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 6]}]: dispatch 2023-07-10T17:14:42.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: audit 2023-07-10T17:14:41.284426+0000 mon.a (mon.0) 692 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [4, 6]}]: dispatch 2023-07-10T17:14:42.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: audit 2023-07-10T17:14:41.284835+0000 mon.a (mon.0) 693 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 6]}]: dispatch 2023-07-10T17:14:42.149 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: audit 2023-07-10T17:14:41.285250+0000 mon.a (mon.0) 694 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 6]}]: dispatch 2023-07-10T17:14:42.149 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: audit 2023-07-10T17:14:41.817406+0000 mon.a (mon.0) 695 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 6]}]': finished 2023-07-10T17:14:42.149 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: audit 2023-07-10T17:14:41.817637+0000 mon.a (mon.0) 696 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2, 3, 0]}]': finished 2023-07-10T17:14:42.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.803471+0000 mon.a (mon.0) 684 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:14:42.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.815060+0000 mon.a (mon.0) 685 : cluster 0 osdmap e108: 8 total, 7 up, 8 in 2023-07-10T17:14:42.151 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.815626+0000 osd.7 (osd.7) 48 : cluster 3 failed to encode map e108 with expected crc 2023-07-10T17:14:42.151 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.820187+0000 osd.7 (osd.7) 49 : cluster 3 failed to encode map e108 with expected crc 2023-07-10T17:14:42.151 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.822085+0000 osd.7 (osd.7) 50 : cluster 3 failed to encode map e108 with expected crc 2023-07-10T17:14:42.151 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.824246+0000 osd.7 (osd.7) 51 : cluster 3 failed to encode map e108 with expected crc 2023-07-10T17:14:42.152 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: cluster 2023-07-10T17:14:40.826658+0000 osd.7 (osd.7) 52 : cluster 3 failed to encode map e108 with expected crc 2023-07-10T17:14:42.152 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: audit 2023-07-10T17:14:41.281701+0000 mon.a (mon.0) 686 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 6]}]: dispatch 2023-07-10T17:14:42.152 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: audit 2023-07-10T17:14:41.282235+0000 mon.a (mon.0) 687 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2, 3, 0]}]: dispatch 2023-07-10T17:14:42.153 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: audit 2023-07-10T17:14:41.282662+0000 mon.a (mon.0) 688 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 6]}]: dispatch 2023-07-10T17:14:42.153 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: audit 2023-07-10T17:14:41.283108+0000 mon.a (mon.0) 689 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2023-07-10T17:14:42.153 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: audit 2023-07-10T17:14:41.283574+0000 mon.a (mon.0) 690 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]: dispatch 2023-07-10T17:14:42.154 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: audit 2023-07-10T17:14:41.284000+0000 mon.a (mon.0) 691 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 6]}]: dispatch 2023-07-10T17:14:42.154 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: audit 2023-07-10T17:14:41.284426+0000 mon.a (mon.0) 692 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [4, 6]}]: dispatch 2023-07-10T17:14:42.154 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: audit 2023-07-10T17:14:41.284835+0000 mon.a (mon.0) 693 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 6]}]: dispatch 2023-07-10T17:14:42.155 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: audit 2023-07-10T17:14:41.285250+0000 mon.a (mon.0) 694 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 6]}]: dispatch 2023-07-10T17:14:42.155 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: audit 2023-07-10T17:14:41.817406+0000 mon.a (mon.0) 695 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 6]}]': finished 2023-07-10T17:14:42.155 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: audit 2023-07-10T17:14:41.817637+0000 mon.a (mon.0) 696 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2, 3, 0]}]': finished 2023-07-10T17:14:42.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: audit 2023-07-10T17:14:41.817831+0000 mon.a (mon.0) 697 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 6]}]': finished 2023-07-10T17:14:42.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: audit 2023-07-10T17:14:41.818016+0000 mon.a (mon.0) 698 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2023-07-10T17:14:42.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: audit 2023-07-10T17:14:41.818198+0000 mon.a (mon.0) 699 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]': finished 2023-07-10T17:14:42.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: audit 2023-07-10T17:14:41.818380+0000 mon.a (mon.0) 700 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 6]}]': finished 2023-07-10T17:14:42.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[63900]: audit 2023-07-10T17:14:41.818544+0000 mon.a (mon.0) 701 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [4, 6]}]': finished 2023-07-10T17:14:42.158 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: audit 2023-07-10T17:14:41.817831+0000 mon.a (mon.0) 697 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 6]}]': finished 2023-07-10T17:14:42.158 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: audit 2023-07-10T17:14:41.818016+0000 mon.a (mon.0) 698 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2023-07-10T17:14:42.158 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: audit 2023-07-10T17:14:41.818198+0000 mon.a (mon.0) 699 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]': finished 2023-07-10T17:14:42.158 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: audit 2023-07-10T17:14:41.818380+0000 mon.a (mon.0) 700 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 6]}]': finished 2023-07-10T17:14:42.159 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:41 smithi116 bash[65980]: audit 2023-07-10T17:14:41.818544+0000 mon.a (mon.0) 701 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [4, 6]}]': finished 2023-07-10T17:14:43.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:42 smithi192 bash[50362]: cluster 2023-07-10T17:14:41.498839+0000 mgr.y (mgr.25390) 282 : cluster 0 pgmap v196: 161 pgs: 13 stale+active+clean, 148 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:43.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:42 smithi192 bash[50362]: audit 2023-07-10T17:14:41.819324+0000 mon.a (mon.0) 702 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 6]}]': finished 2023-07-10T17:14:43.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:42 smithi192 bash[50362]: cluster 2023-07-10T17:14:41.819352+0000 osd.7 (osd.7) 53 : cluster 3 failed to encode map e109 with expected crc 2023-07-10T17:14:43.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:42 smithi192 bash[50362]: audit 2023-07-10T17:14:41.819560+0000 mon.a (mon.0) 703 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 6]}]': finished 2023-07-10T17:14:43.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:42 smithi192 bash[50362]: cluster 2023-07-10T17:14:41.819741+0000 mon.a (mon.0) 704 : cluster 0 osdmap e109: 8 total, 7 up, 8 in 2023-07-10T17:14:43.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:42 smithi192 bash[50362]: cluster 2023-07-10T17:14:41.823345+0000 osd.7 (osd.7) 54 : cluster 3 failed to encode map e109 with expected crc 2023-07-10T17:14:43.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:42 smithi192 bash[50362]: cluster 2023-07-10T17:14:41.824918+0000 osd.7 (osd.7) 55 : cluster 3 failed to encode map e109 with expected crc 2023-07-10T17:14:43.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:42 smithi192 bash[50362]: cluster 2023-07-10T17:14:41.826541+0000 osd.7 (osd.7) 56 : cluster 3 failed to encode map e109 with expected crc 2023-07-10T17:14:43.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:42 smithi192 bash[50362]: cluster 2023-07-10T17:14:41.828917+0000 osd.7 (osd.7) 57 : cluster 3 failed to encode map e109 with expected crc 2023-07-10T17:14:43.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:42 smithi192 bash[50362]: cluster 2023-07-10T17:14:42.611090+0000 mon.a (mon.0) 705 : cluster 0 osdmap e110: 8 total, 7 up, 8 in 2023-07-10T17:14:43.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[63900]: cluster 2023-07-10T17:14:41.498839+0000 mgr.y (mgr.25390) 282 : cluster 0 pgmap v196: 161 pgs: 13 stale+active+clean, 148 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:43.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[63900]: audit 2023-07-10T17:14:41.819324+0000 mon.a (mon.0) 702 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 6]}]': finished 2023-07-10T17:14:43.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[63900]: cluster 2023-07-10T17:14:41.819352+0000 osd.7 (osd.7) 53 : cluster 3 failed to encode map e109 with expected crc 2023-07-10T17:14:43.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[63900]: audit 2023-07-10T17:14:41.819560+0000 mon.a (mon.0) 703 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 6]}]': finished 2023-07-10T17:14:43.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[63900]: cluster 2023-07-10T17:14:41.819741+0000 mon.a (mon.0) 704 : cluster 0 osdmap e109: 8 total, 7 up, 8 in 2023-07-10T17:14:43.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[63900]: cluster 2023-07-10T17:14:41.823345+0000 osd.7 (osd.7) 54 : cluster 3 failed to encode map e109 with expected crc 2023-07-10T17:14:43.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[63900]: cluster 2023-07-10T17:14:41.824918+0000 osd.7 (osd.7) 55 : cluster 3 failed to encode map e109 with expected crc 2023-07-10T17:14:43.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[63900]: cluster 2023-07-10T17:14:41.826541+0000 osd.7 (osd.7) 56 : cluster 3 failed to encode map e109 with expected crc 2023-07-10T17:14:43.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[63900]: cluster 2023-07-10T17:14:41.828917+0000 osd.7 (osd.7) 57 : cluster 3 failed to encode map e109 with expected crc 2023-07-10T17:14:43.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[63900]: cluster 2023-07-10T17:14:42.611090+0000 mon.a (mon.0) 705 : cluster 0 osdmap e110: 8 total, 7 up, 8 in 2023-07-10T17:14:43.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[65980]: cluster 2023-07-10T17:14:41.498839+0000 mgr.y (mgr.25390) 282 : cluster 0 pgmap v196: 161 pgs: 13 stale+active+clean, 148 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:43.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[65980]: audit 2023-07-10T17:14:41.819324+0000 mon.a (mon.0) 702 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 6]}]': finished 2023-07-10T17:14:43.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[65980]: cluster 2023-07-10T17:14:41.819352+0000 osd.7 (osd.7) 53 : cluster 3 failed to encode map e109 with expected crc 2023-07-10T17:14:43.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[65980]: audit 2023-07-10T17:14:41.819560+0000 mon.a (mon.0) 703 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 6]}]': finished 2023-07-10T17:14:43.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[65980]: cluster 2023-07-10T17:14:41.819741+0000 mon.a (mon.0) 704 : cluster 0 osdmap e109: 8 total, 7 up, 8 in 2023-07-10T17:14:43.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[65980]: cluster 2023-07-10T17:14:41.823345+0000 osd.7 (osd.7) 54 : cluster 3 failed to encode map e109 with expected crc 2023-07-10T17:14:43.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[65980]: cluster 2023-07-10T17:14:41.824918+0000 osd.7 (osd.7) 55 : cluster 3 failed to encode map e109 with expected crc 2023-07-10T17:14:43.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[65980]: cluster 2023-07-10T17:14:41.826541+0000 osd.7 (osd.7) 56 : cluster 3 failed to encode map e109 with expected crc 2023-07-10T17:14:43.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[65980]: cluster 2023-07-10T17:14:41.828917+0000 osd.7 (osd.7) 57 : cluster 3 failed to encode map e109 with expected crc 2023-07-10T17:14:43.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:42 smithi116 bash[65980]: cluster 2023-07-10T17:14:42.611090+0000 mon.a (mon.0) 705 : cluster 0 osdmap e110: 8 total, 7 up, 8 in 2023-07-10T17:14:43.702 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:43 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.6.service: Succeeded. 2023-07-10T17:14:43.703 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:43 smithi192 systemd[1]: Stopped Ceph osd.6 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:14:43.703 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:43 smithi192 systemd[1]: Started Ceph osd.6 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:14:44.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:43 smithi192 bash[50362]: cluster 2023-07-10T17:14:42.606093+0000 osd.7 (osd.7) 58 : cluster 3 failed to encode map e110 with expected crc 2023-07-10T17:14:44.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:43 smithi192 bash[50362]: cluster 2023-07-10T17:14:43.613012+0000 mon.a (mon.0) 706 : cluster 0 osdmap e111: 8 total, 7 up, 8 in 2023-07-10T17:14:44.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:43 smithi192 bash[50362]: audit 2023-07-10T17:14:43.674100+0000 mon.a (mon.0) 707 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:44.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:43 smithi192 bash[50362]: audit 2023-07-10T17:14:43.685961+0000 mon.a (mon.0) 708 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:44.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:43 smithi116 bash[63900]: cluster 2023-07-10T17:14:42.606093+0000 osd.7 (osd.7) 58 : cluster 3 failed to encode map e110 with expected crc 2023-07-10T17:14:44.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:43 smithi116 bash[63900]: cluster 2023-07-10T17:14:43.613012+0000 mon.a (mon.0) 706 : cluster 0 osdmap e111: 8 total, 7 up, 8 in 2023-07-10T17:14:44.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:43 smithi116 bash[63900]: audit 2023-07-10T17:14:43.674100+0000 mon.a (mon.0) 707 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:44.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:43 smithi116 bash[63900]: audit 2023-07-10T17:14:43.685961+0000 mon.a (mon.0) 708 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:44.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:43 smithi116 bash[65980]: cluster 2023-07-10T17:14:42.606093+0000 osd.7 (osd.7) 58 : cluster 3 failed to encode map e110 with expected crc 2023-07-10T17:14:44.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:43 smithi116 bash[65980]: cluster 2023-07-10T17:14:43.613012+0000 mon.a (mon.0) 706 : cluster 0 osdmap e111: 8 total, 7 up, 8 in 2023-07-10T17:14:44.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:43 smithi116 bash[65980]: audit 2023-07-10T17:14:43.674100+0000 mon.a (mon.0) 707 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:44.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:43 smithi116 bash[65980]: audit 2023-07-10T17:14:43.685961+0000 mon.a (mon.0) 708 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:45.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:44 smithi192 bash[50362]: cluster 2023-07-10T17:14:43.499783+0000 mgr.y (mgr.25390) 283 : cluster 0 pgmap v199: 161 pgs: 2 unknown, 12 active+undersized, 8 active+undersized+degraded, 3 peering, 7 stale+active+clean, 129 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 44/999 objects degraded (4.404%) 2023-07-10T17:14:45.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:44 smithi192 bash[50362]: cluster 2023-07-10T17:14:43.838600+0000 mon.a (mon.0) 709 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-07-10T17:14:45.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:44 smithi192 bash[50362]: cluster 2023-07-10T17:14:43.838656+0000 mon.a (mon.0) 710 : cluster 3 Health check failed: Degraded data redundancy: 44/999 objects degraded (4.404%), 8 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:45.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:44 smithi192 bash[50362]: cluster 2023-07-10T17:14:43.855227+0000 osd.7 (osd.7) 59 : cluster 3 failed to encode map e111 with expected crc 2023-07-10T17:14:45.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:44 smithi116 bash[63900]: cluster 2023-07-10T17:14:43.499783+0000 mgr.y (mgr.25390) 283 : cluster 0 pgmap v199: 161 pgs: 2 unknown, 12 active+undersized, 8 active+undersized+degraded, 3 peering, 7 stale+active+clean, 129 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 44/999 objects degraded (4.404%) 2023-07-10T17:14:45.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:44 smithi116 bash[63900]: cluster 2023-07-10T17:14:43.838600+0000 mon.a (mon.0) 709 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-07-10T17:14:45.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:44 smithi116 bash[63900]: cluster 2023-07-10T17:14:43.838656+0000 mon.a (mon.0) 710 : cluster 3 Health check failed: Degraded data redundancy: 44/999 objects degraded (4.404%), 8 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:45.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:44 smithi116 bash[63900]: cluster 2023-07-10T17:14:43.855227+0000 osd.7 (osd.7) 59 : cluster 3 failed to encode map e111 with expected crc 2023-07-10T17:14:45.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:44 smithi116 bash[65980]: cluster 2023-07-10T17:14:43.499783+0000 mgr.y (mgr.25390) 283 : cluster 0 pgmap v199: 161 pgs: 2 unknown, 12 active+undersized, 8 active+undersized+degraded, 3 peering, 7 stale+active+clean, 129 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 44/999 objects degraded (4.404%) 2023-07-10T17:14:45.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:44 smithi116 bash[65980]: cluster 2023-07-10T17:14:43.838600+0000 mon.a (mon.0) 709 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-07-10T17:14:45.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:44 smithi116 bash[65980]: cluster 2023-07-10T17:14:43.838656+0000 mon.a (mon.0) 710 : cluster 3 Health check failed: Degraded data redundancy: 44/999 objects degraded (4.404%), 8 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:45.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:44 smithi116 bash[65980]: cluster 2023-07-10T17:14:43.855227+0000 osd.7 (osd.7) 59 : cluster 3 failed to encode map e111 with expected crc 2023-07-10T17:14:45.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:45 smithi192 bash[57092]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-07-10T17:14:45.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:45 smithi192 bash[57092]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2023-07-10T17:14:45.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:45 smithi192 bash[57092]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2023-07-10T17:14:45.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:45 smithi192 bash[57092]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2023-07-10T17:14:45.888 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:45 smithi192 bash[57092]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2023-07-10T17:14:45.888 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:45 smithi192 bash[57092]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-07-10T17:14:45.888 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:45 smithi192 bash[57092]: --> ceph-volume raw activate successful for osd ID: 6 2023-07-10T17:14:47.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:46 smithi192 bash[50362]: cluster 2023-07-10T17:14:45.501019+0000 mgr.y (mgr.25390) 284 : cluster 0 pgmap v201: 161 pgs: 1 active+clean+remapped, 2 unknown, 22 active+undersized, 12 active+undersized+degraded, 3 peering, 1 stale+active+clean, 120 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 77/999 objects degraded (7.708%); 1/999 objects misplaced (0.100%); 0 B/s, 0 objects/s recovering 2023-07-10T17:14:47.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:46 smithi116 bash[63900]: cluster 2023-07-10T17:14:45.501019+0000 mgr.y (mgr.25390) 284 : cluster 0 pgmap v201: 161 pgs: 1 active+clean+remapped, 2 unknown, 22 active+undersized, 12 active+undersized+degraded, 3 peering, 1 stale+active+clean, 120 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 77/999 objects degraded (7.708%); 1/999 objects misplaced (0.100%); 0 B/s, 0 objects/s recovering 2023-07-10T17:14:47.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:46 smithi116 bash[65980]: cluster 2023-07-10T17:14:45.501019+0000 mgr.y (mgr.25390) 284 : cluster 0 pgmap v201: 161 pgs: 1 active+clean+remapped, 2 unknown, 22 active+undersized, 12 active+undersized+degraded, 3 peering, 1 stale+active+clean, 120 active+clean; 604 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 77/999 objects degraded (7.708%); 1/999 objects misplaced (0.100%); 0 B/s, 0 objects/s recovering 2023-07-10T17:14:47.637 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:47 smithi192 bash[57401]: debug 2023-07-10T17:14:47.204+0000 7fb6d33d4700 1 -- 172.21.15.192:0/3453687580 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5588a28524e0 con 0x5588a28ad000 2023-07-10T17:14:48.137 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:47 smithi192 bash[57401]: debug 2023-07-10T17:14:47.744+0000 7fb6dc559640 -1 Falling back to public interface 2023-07-10T17:14:48.614 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:14:48.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:48 smithi192 bash[50362]: cluster 2023-07-10T17:14:47.502806+0000 mgr.y (mgr.25390) 285 : cluster 0 pgmap v202: 161 pgs: 8 active+clean+remapped, 34 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 123/999 objects degraded (12.312%); 1/999 objects misplaced (0.100%); 0 B/s, 0 objects/s recovering 2023-07-10T17:14:48.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:48 smithi192 bash[50362]: cluster 2023-07-10T17:14:47.855237+0000 mon.a (mon.0) 711 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-07-10T17:14:48.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:48 smithi116 bash[63900]: cluster 2023-07-10T17:14:47.502806+0000 mgr.y (mgr.25390) 285 : cluster 0 pgmap v202: 161 pgs: 8 active+clean+remapped, 34 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 123/999 objects degraded (12.312%); 1/999 objects misplaced (0.100%); 0 B/s, 0 objects/s recovering 2023-07-10T17:14:48.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:48 smithi116 bash[63900]: cluster 2023-07-10T17:14:47.855237+0000 mon.a (mon.0) 711 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-07-10T17:14:48.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:48 smithi116 bash[65980]: cluster 2023-07-10T17:14:47.502806+0000 mgr.y (mgr.25390) 285 : cluster 0 pgmap v202: 161 pgs: 8 active+clean+remapped, 34 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 123/999 objects degraded (12.312%); 1/999 objects misplaced (0.100%); 0 B/s, 0 objects/s recovering 2023-07-10T17:14:48.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:48 smithi116 bash[65980]: cluster 2023-07-10T17:14:47.855237+0000 mon.a (mon.0) 711 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-07-10T17:14:49.352 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:14:49.353 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (7m) 83s ago 14m 10.9M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:14:49.353 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (6m) 26s ago 13m 25.9M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:14:49.353 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 error 83s ago 11m - - 2023-07-10T17:14:49.353 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (5m) 26s ago 18m 429M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:14:49.353 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (6m) 83s ago 20m 475M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:14:49.353 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (5m) 83s ago 20m 59.6M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:14:49.353 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (4m) 26s ago 18m 53.1M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 74a082a2e795 2023-07-10T17:14:49.354 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (4m) 83s ago 18m 49.9M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:14:49.354 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (8m) 83s ago 14m 9371k - 0.18.1 e5a616e4b9cf 3bd058d91353 2023-07-10T17:14:49.354 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (7m) 26s ago 14m 3808k - 0.18.1 e5a616e4b9cf 63551615c679 2023-07-10T17:14:49.354 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (2m) 83s ago 18m 53.2M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 0562f587fbca 2023-07-10T17:14:49.354 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (2m) 83s ago 17m 51.6M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 5b2a12a0bfc9 2023-07-10T17:14:49.355 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (118s) 83s ago 17m 47.9M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 88096bc0faa9 2023-07-10T17:14:49.355 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (88s) 83s ago 16m - 4096M 18.1.2-30-g8507faa7 17e8f0d79247 61ae280c471b 2023-07-10T17:14:49.355 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (59s) 26s ago 16m 51.7M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 44d809c18118 2023-07-10T17:14:49.355 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (31s) 26s ago 16m - 4096M 18.1.2-30-g8507faa7 17e8f0d79247 179270eb7044 2023-07-10T17:14:49.355 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 starting - - - 4096M 2023-07-10T17:14:49.356 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (15m) 26s ago 15m 65.3M 4096M 16.2.0 24ecd6d5f14c 354515ef92ea 2023-07-10T17:14:49.356 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 error 26s ago 14m - - 2023-07-10T17:14:49.356 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (12m) 83s ago 12m 55.9M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:14:49.356 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (12m) 26s ago 12m 56.1M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:14:49.356 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (12m) 83s ago 12m 55.5M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:14:49.357 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (11m) 26s ago 11m 57.5M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:14:49.788 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:14:49.789 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:14:49.789 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:14:49.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:14:49.789 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:14:49.789 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:14:49.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:14:49.790 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:14:49.790 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2023-07-10T17:14:49.790 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 6 2023-07-10T17:14:49.790 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:14:49.791 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:14:49.791 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:14:49.791 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:14:49.791 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:14:49.792 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 5, 2023-07-10T17:14:49.792 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 11 2023-07-10T17:14:49.792 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:14:49.792 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:14:49.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:49 smithi192 bash[50362]: audit 2023-07-10T17:14:48.594205+0000 mgr.y (mgr.25390) 286 : audit 0 from='client.44524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:49.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:49 smithi192 bash[50362]: audit 2023-07-10T17:14:48.960865+0000 mgr.y (mgr.25390) 287 : audit 0 from='client.34674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:49.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:49 smithi116 bash[63900]: audit 2023-07-10T17:14:48.594205+0000 mgr.y (mgr.25390) 286 : audit 0 from='client.44524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:49.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:49 smithi116 bash[63900]: audit 2023-07-10T17:14:48.960865+0000 mgr.y (mgr.25390) 287 : audit 0 from='client.34674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:49.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:49 smithi116 bash[65980]: audit 2023-07-10T17:14:48.594205+0000 mgr.y (mgr.25390) 286 : audit 0 from='client.44524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:49.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:49 smithi116 bash[65980]: audit 2023-07-10T17:14:48.960865+0000 mgr.y (mgr.25390) 287 : audit 0 from='client.34674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:50.152 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:14:50.152 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:14:50.152 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:14:50.153 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:14:50.153 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:14:50.153 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2023-07-10T17:14:50.153 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2023-07-10T17:14:50.153 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:14:50.153 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "11/23 daemons upgraded", 2023-07-10T17:14:50.153 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading osd daemons", 2023-07-10T17:14:50.153 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:14:50.154 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:14:50.560 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 2 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 123/999 objects degraded (12.312%), 17 pgs degraded 2023-07-10T17:14:50.561 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:14:50.561 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:14:50.561 INFO:teuthology.orchestra.run.smithi116.stdout: daemon prometheus.a on smithi192 is in error state 2023-07-10T17:14:50.561 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] OSD_DOWN: 1 osds down 2023-07-10T17:14:50.561 INFO:teuthology.orchestra.run.smithi116.stdout: osd.6 (root=default,host=smithi192) is down 2023-07-10T17:14:50.561 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 123/999 objects degraded (12.312%), 17 pgs degraded 2023-07-10T17:14:50.562 INFO:teuthology.orchestra.run.smithi116.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2023-07-10T17:14:50.562 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2023-07-10T17:14:50.562 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.1a is active+undersized+degraded, acting [4,7] 2023-07-10T17:14:50.562 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.1d is active+undersized+degraded, acting [7,0] 2023-07-10T17:14:50.562 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2023-07-10T17:14:50.562 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2023-07-10T17:14:50.562 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2023-07-10T17:14:50.563 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2023-07-10T17:14:50.563 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2023-07-10T17:14:50.563 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2023-07-10T17:14:50.563 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2023-07-10T17:14:50.563 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2023-07-10T17:14:50.564 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2023-07-10T17:14:50.564 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2023-07-10T17:14:50.564 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2023-07-10T17:14:50.565 INFO:teuthology.orchestra.run.smithi116.stdout: pg 4.1f is active+undersized+degraded, acting [5,1] 2023-07-10T17:14:50.565 INFO:teuthology.orchestra.run.smithi116.stdout: pg 6.c is active+undersized+degraded, acting [3,5] 2023-07-10T17:14:50.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:50 smithi192 bash[57401]: debug 2023-07-10T17:14:50.828+0000 7fb6dc559640 -1 osd.6 107 log_to_monitors true 2023-07-10T17:14:50.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:50 smithi192 bash[50362]: audit 2023-07-10T17:14:49.323271+0000 mgr.y (mgr.25390) 288 : audit 0 from='client.44536 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:50.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:50 smithi192 bash[50362]: cluster 2023-07-10T17:14:49.503511+0000 mgr.y (mgr.25390) 289 : cluster 0 pgmap v203: 161 pgs: 8 active+clean+remapped, 34 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 123/999 objects degraded (12.312%); 1/999 objects misplaced (0.100%); 0 B/s, 0 objects/s recovering 2023-07-10T17:14:50.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:50 smithi192 bash[50362]: audit 2023-07-10T17:14:49.783720+0000 mon.c (mon.1) 22 : audit 0 from='client.44542 172.21.15.116:0/3541431506' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:50.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:50 smithi192 bash[50362]: audit 2023-07-10T17:14:50.146816+0000 mgr.y (mgr.25390) 290 : audit 0 from='client.34692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:50.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:50 smithi192 bash[50362]: audit 2023-07-10T17:14:50.556028+0000 mon.a (mon.0) 712 : audit 0 from='client.34698 172.21.15.116:0/3903812563' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:14:50.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:50 smithi116 bash[63900]: audit 2023-07-10T17:14:49.323271+0000 mgr.y (mgr.25390) 288 : audit 0 from='client.44536 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:50.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:50 smithi116 bash[63900]: cluster 2023-07-10T17:14:49.503511+0000 mgr.y (mgr.25390) 289 : cluster 0 pgmap v203: 161 pgs: 8 active+clean+remapped, 34 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 123/999 objects degraded (12.312%); 1/999 objects misplaced (0.100%); 0 B/s, 0 objects/s recovering 2023-07-10T17:14:50.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:50 smithi116 bash[63900]: audit 2023-07-10T17:14:49.783720+0000 mon.c (mon.1) 22 : audit 0 from='client.44542 172.21.15.116:0/3541431506' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:50.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:50 smithi116 bash[63900]: audit 2023-07-10T17:14:50.146816+0000 mgr.y (mgr.25390) 290 : audit 0 from='client.34692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:50.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:50 smithi116 bash[63900]: audit 2023-07-10T17:14:50.556028+0000 mon.a (mon.0) 712 : audit 0 from='client.34698 172.21.15.116:0/3903812563' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:14:50.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:50 smithi116 bash[65980]: audit 2023-07-10T17:14:49.323271+0000 mgr.y (mgr.25390) 288 : audit 0 from='client.44536 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:50.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:50 smithi116 bash[65980]: cluster 2023-07-10T17:14:49.503511+0000 mgr.y (mgr.25390) 289 : cluster 0 pgmap v203: 161 pgs: 8 active+clean+remapped, 34 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 123/999 objects degraded (12.312%); 1/999 objects misplaced (0.100%); 0 B/s, 0 objects/s recovering 2023-07-10T17:14:50.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:50 smithi116 bash[65980]: audit 2023-07-10T17:14:49.783720+0000 mon.c (mon.1) 22 : audit 0 from='client.44542 172.21.15.116:0/3541431506' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:14:50.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:50 smithi116 bash[65980]: audit 2023-07-10T17:14:50.146816+0000 mgr.y (mgr.25390) 290 : audit 0 from='client.34692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:14:50.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:50 smithi116 bash[65980]: audit 2023-07-10T17:14:50.556028+0000 mon.a (mon.0) 712 : audit 0 from='client.34698 172.21.15.116:0/3903812563' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:14:51.889 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:14:51 smithi192 bash[57401]: debug 2023-07-10T17:14:51.684+0000 7fb6cc1c3700 -1 osd.6 107 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-10T17:14:51.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:51 smithi192 bash[50362]: audit 2023-07-10T17:14:50.834680+0000 mon.a (mon.0) 713 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-10T17:14:51.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:51 smithi192 bash[50362]: audit 2023-07-10T17:14:50.835318+0000 mon.b (mon.2) 5 : audit 1 from='osd.6 [v2:172.21.15.192:6816/3392209684,v1:172.21.15.192:6817/3392209684]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-10T17:14:51.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:51 smithi192 bash[50362]: audit 2023-07-10T17:14:51.157473+0000 mon.a (mon.0) 714 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:51.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:51 smithi192 bash[50362]: audit 2023-07-10T17:14:51.166871+0000 mon.a (mon.0) 715 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:51.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:51 smithi116 bash[63900]: audit 2023-07-10T17:14:50.834680+0000 mon.a (mon.0) 713 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-10T17:14:51.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:51 smithi116 bash[63900]: audit 2023-07-10T17:14:50.835318+0000 mon.b (mon.2) 5 : audit 1 from='osd.6 [v2:172.21.15.192:6816/3392209684,v1:172.21.15.192:6817/3392209684]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-10T17:14:51.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:51 smithi116 bash[63900]: audit 2023-07-10T17:14:51.157473+0000 mon.a (mon.0) 714 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:51.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:51 smithi116 bash[63900]: audit 2023-07-10T17:14:51.166871+0000 mon.a (mon.0) 715 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:51.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:51 smithi116 bash[65980]: audit 2023-07-10T17:14:50.834680+0000 mon.a (mon.0) 713 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-10T17:14:51.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:51 smithi116 bash[65980]: audit 2023-07-10T17:14:50.835318+0000 mon.b (mon.2) 5 : audit 1 from='osd.6 [v2:172.21.15.192:6816/3392209684,v1:172.21.15.192:6817/3392209684]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-10T17:14:51.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:51 smithi116 bash[65980]: audit 2023-07-10T17:14:51.157473+0000 mon.a (mon.0) 714 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:51.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:51 smithi116 bash[65980]: audit 2023-07-10T17:14:51.166871+0000 mon.a (mon.0) 715 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:53.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:52 smithi192 bash[50362]: cluster 2023-07-10T17:14:51.504851+0000 mgr.y (mgr.25390) 291 : cluster 0 pgmap v204: 161 pgs: 8 active+clean+remapped, 34 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 123/999 objects degraded (12.312%); 1/999 objects misplaced (0.100%); 0 B/s, 0 objects/s recovering 2023-07-10T17:14:53.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:52 smithi192 bash[50362]: cluster 2023-07-10T17:14:51.637710+0000 osd.7 (osd.7) 60 : cluster 3 failed to encode map e112 with expected crc 2023-07-10T17:14:53.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:52 smithi192 bash[50362]: audit 2023-07-10T17:14:51.637761+0000 mon.a (mon.0) 716 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-10T17:14:53.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:52 smithi192 bash[50362]: cluster 2023-07-10T17:14:51.637867+0000 mon.a (mon.0) 717 : cluster 0 osdmap e112: 8 total, 7 up, 8 in 2023-07-10T17:14:53.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:52 smithi192 bash[50362]: audit 2023-07-10T17:14:51.644163+0000 mon.a (mon.0) 718 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:14:53.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:52 smithi192 bash[50362]: audit 2023-07-10T17:14:51.644686+0000 mon.b (mon.2) 6 : audit 1 from='osd.6 [v2:172.21.15.192:6816/3392209684,v1:172.21.15.192:6817/3392209684]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:14:53.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:52 smithi192 bash[50362]: audit 2023-07-10T17:14:51.975068+0000 mon.a (mon.0) 719 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:53.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:52 smithi192 bash[50362]: audit 2023-07-10T17:14:51.989182+0000 mon.a (mon.0) 720 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:53.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:52 smithi192 bash[50362]: cluster 2023-07-10T17:14:52.604499+0000 mon.a (mon.0) 721 : cluster 3 Health check update: Degraded data redundancy: 123/999 objects degraded (12.312%), 17 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:53.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[65980]: cluster 2023-07-10T17:14:51.504851+0000 mgr.y (mgr.25390) 291 : cluster 0 pgmap v204: 161 pgs: 8 active+clean+remapped, 34 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 123/999 objects degraded (12.312%); 1/999 objects misplaced (0.100%); 0 B/s, 0 objects/s recovering 2023-07-10T17:14:53.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[65980]: cluster 2023-07-10T17:14:51.637710+0000 osd.7 (osd.7) 60 : cluster 3 failed to encode map e112 with expected crc 2023-07-10T17:14:53.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[65980]: audit 2023-07-10T17:14:51.637761+0000 mon.a (mon.0) 716 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-10T17:14:53.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[65980]: cluster 2023-07-10T17:14:51.637867+0000 mon.a (mon.0) 717 : cluster 0 osdmap e112: 8 total, 7 up, 8 in 2023-07-10T17:14:53.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[65980]: audit 2023-07-10T17:14:51.644163+0000 mon.a (mon.0) 718 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:14:53.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[65980]: audit 2023-07-10T17:14:51.644686+0000 mon.b (mon.2) 6 : audit 1 from='osd.6 [v2:172.21.15.192:6816/3392209684,v1:172.21.15.192:6817/3392209684]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:14:53.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[65980]: audit 2023-07-10T17:14:51.975068+0000 mon.a (mon.0) 719 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:53.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[65980]: audit 2023-07-10T17:14:51.989182+0000 mon.a (mon.0) 720 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:53.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[65980]: cluster 2023-07-10T17:14:52.604499+0000 mon.a (mon.0) 721 : cluster 3 Health check update: Degraded data redundancy: 123/999 objects degraded (12.312%), 17 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:53.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[63900]: cluster 2023-07-10T17:14:51.504851+0000 mgr.y (mgr.25390) 291 : cluster 0 pgmap v204: 161 pgs: 8 active+clean+remapped, 34 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 123/999 objects degraded (12.312%); 1/999 objects misplaced (0.100%); 0 B/s, 0 objects/s recovering 2023-07-10T17:14:53.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[63900]: cluster 2023-07-10T17:14:51.637710+0000 osd.7 (osd.7) 60 : cluster 3 failed to encode map e112 with expected crc 2023-07-10T17:14:53.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[63900]: audit 2023-07-10T17:14:51.637761+0000 mon.a (mon.0) 716 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-10T17:14:53.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[63900]: cluster 2023-07-10T17:14:51.637867+0000 mon.a (mon.0) 717 : cluster 0 osdmap e112: 8 total, 7 up, 8 in 2023-07-10T17:14:53.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[63900]: audit 2023-07-10T17:14:51.644163+0000 mon.a (mon.0) 718 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:14:53.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[63900]: audit 2023-07-10T17:14:51.644686+0000 mon.b (mon.2) 6 : audit 1 from='osd.6 [v2:172.21.15.192:6816/3392209684,v1:172.21.15.192:6817/3392209684]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:14:53.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[63900]: audit 2023-07-10T17:14:51.975068+0000 mon.a (mon.0) 719 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:53.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[63900]: audit 2023-07-10T17:14:51.989182+0000 mon.a (mon.0) 720 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:53.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:52 smithi116 bash[63900]: cluster 2023-07-10T17:14:52.604499+0000 mon.a (mon.0) 721 : cluster 3 Health check update: Degraded data redundancy: 123/999 objects degraded (12.312%), 17 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:54.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:53 smithi192 bash[50362]: cluster 2023-07-10T17:14:52.636279+0000 mon.a (mon.0) 722 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:14:54.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:53 smithi192 bash[50362]: cluster 2023-07-10T17:14:52.644870+0000 osd.7 (osd.7) 61 : cluster 3 failed to encode map e113 with expected crc 2023-07-10T17:14:54.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:53 smithi192 bash[50362]: cluster 2023-07-10T17:14:52.649055+0000 mon.a (mon.0) 723 : cluster 1 osd.6 [v2:172.21.15.192:6816/3392209684,v1:172.21.15.192:6817/3392209684] boot 2023-07-10T17:14:54.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:53 smithi192 bash[50362]: cluster 2023-07-10T17:14:52.649343+0000 mon.a (mon.0) 724 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2023-07-10T17:14:54.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:53 smithi192 bash[50362]: audit 2023-07-10T17:14:52.650928+0000 mon.a (mon.0) 725 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T17:14:54.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:53 smithi192 bash[50362]: cluster 2023-07-10T17:14:52.652092+0000 osd.7 (osd.7) 62 : cluster 3 failed to encode map e113 with expected crc 2023-07-10T17:14:54.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:53 smithi116 bash[63900]: cluster 2023-07-10T17:14:52.636279+0000 mon.a (mon.0) 722 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:14:54.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:53 smithi116 bash[63900]: cluster 2023-07-10T17:14:52.644870+0000 osd.7 (osd.7) 61 : cluster 3 failed to encode map e113 with expected crc 2023-07-10T17:14:54.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:53 smithi116 bash[63900]: cluster 2023-07-10T17:14:52.649055+0000 mon.a (mon.0) 723 : cluster 1 osd.6 [v2:172.21.15.192:6816/3392209684,v1:172.21.15.192:6817/3392209684] boot 2023-07-10T17:14:54.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:53 smithi116 bash[63900]: cluster 2023-07-10T17:14:52.649343+0000 mon.a (mon.0) 724 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2023-07-10T17:14:54.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:53 smithi116 bash[63900]: audit 2023-07-10T17:14:52.650928+0000 mon.a (mon.0) 725 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T17:14:54.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:53 smithi116 bash[63900]: cluster 2023-07-10T17:14:52.652092+0000 osd.7 (osd.7) 62 : cluster 3 failed to encode map e113 with expected crc 2023-07-10T17:14:54.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:53 smithi116 bash[65980]: cluster 2023-07-10T17:14:52.636279+0000 mon.a (mon.0) 722 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:14:54.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:53 smithi116 bash[65980]: cluster 2023-07-10T17:14:52.644870+0000 osd.7 (osd.7) 61 : cluster 3 failed to encode map e113 with expected crc 2023-07-10T17:14:54.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:53 smithi116 bash[65980]: cluster 2023-07-10T17:14:52.649055+0000 mon.a (mon.0) 723 : cluster 1 osd.6 [v2:172.21.15.192:6816/3392209684,v1:172.21.15.192:6817/3392209684] boot 2023-07-10T17:14:54.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:53 smithi116 bash[65980]: cluster 2023-07-10T17:14:52.649343+0000 mon.a (mon.0) 724 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2023-07-10T17:14:54.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:53 smithi116 bash[65980]: audit 2023-07-10T17:14:52.650928+0000 mon.a (mon.0) 725 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-10T17:14:54.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:53 smithi116 bash[65980]: cluster 2023-07-10T17:14:52.652092+0000 osd.7 (osd.7) 62 : cluster 3 failed to encode map e113 with expected crc 2023-07-10T17:14:55.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:54 smithi192 bash[50362]: cluster 2023-07-10T17:14:53.505891+0000 mgr.y (mgr.25390) 292 : cluster 0 pgmap v207: 161 pgs: 8 active+clean+remapped, 34 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 123/999 objects degraded (12.312%); 1/999 objects misplaced (0.100%) 2023-07-10T17:14:55.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:54 smithi192 bash[50362]: cluster 2023-07-10T17:14:53.665873+0000 mon.a (mon.0) 726 : cluster 0 osdmap e114: 8 total, 8 up, 8 in 2023-07-10T17:14:55.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:54 smithi192 bash[50362]: cluster 2023-07-10T17:14:53.668484+0000 osd.7 (osd.7) 63 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:54 smithi192 bash[50362]: cluster 2023-07-10T17:14:53.675386+0000 osd.7 (osd.7) 64 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:54 smithi192 bash[50362]: cluster 2023-07-10T17:14:53.677430+0000 osd.7 (osd.7) 65 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:54 smithi192 bash[50362]: cluster 2023-07-10T17:14:53.679424+0000 osd.7 (osd.7) 66 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:54 smithi192 bash[50362]: cluster 2023-07-10T17:14:53.680938+0000 osd.7 (osd.7) 67 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:54 smithi192 bash[50362]: cluster 2023-07-10T17:14:53.682732+0000 osd.7 (osd.7) 68 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:54 smithi192 bash[50362]: cluster 2023-07-10T17:14:54.665746+0000 mon.a (mon.0) 727 : cluster 0 osdmap e115: 8 total, 8 up, 8 in 2023-07-10T17:14:55.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[63900]: cluster 2023-07-10T17:14:53.505891+0000 mgr.y (mgr.25390) 292 : cluster 0 pgmap v207: 161 pgs: 8 active+clean+remapped, 34 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 123/999 objects degraded (12.312%); 1/999 objects misplaced (0.100%) 2023-07-10T17:14:55.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[63900]: cluster 2023-07-10T17:14:53.665873+0000 mon.a (mon.0) 726 : cluster 0 osdmap e114: 8 total, 8 up, 8 in 2023-07-10T17:14:55.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[63900]: cluster 2023-07-10T17:14:53.668484+0000 osd.7 (osd.7) 63 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[63900]: cluster 2023-07-10T17:14:53.675386+0000 osd.7 (osd.7) 64 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[63900]: cluster 2023-07-10T17:14:53.677430+0000 osd.7 (osd.7) 65 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[63900]: cluster 2023-07-10T17:14:53.679424+0000 osd.7 (osd.7) 66 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[63900]: cluster 2023-07-10T17:14:53.680938+0000 osd.7 (osd.7) 67 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[63900]: cluster 2023-07-10T17:14:53.682732+0000 osd.7 (osd.7) 68 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[63900]: cluster 2023-07-10T17:14:54.665746+0000 mon.a (mon.0) 727 : cluster 0 osdmap e115: 8 total, 8 up, 8 in 2023-07-10T17:14:55.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[65980]: cluster 2023-07-10T17:14:53.505891+0000 mgr.y (mgr.25390) 292 : cluster 0 pgmap v207: 161 pgs: 8 active+clean+remapped, 34 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 123/999 objects degraded (12.312%); 1/999 objects misplaced (0.100%) 2023-07-10T17:14:55.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[65980]: cluster 2023-07-10T17:14:53.665873+0000 mon.a (mon.0) 726 : cluster 0 osdmap e114: 8 total, 8 up, 8 in 2023-07-10T17:14:55.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[65980]: cluster 2023-07-10T17:14:53.668484+0000 osd.7 (osd.7) 63 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[65980]: cluster 2023-07-10T17:14:53.675386+0000 osd.7 (osd.7) 64 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[65980]: cluster 2023-07-10T17:14:53.677430+0000 osd.7 (osd.7) 65 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[65980]: cluster 2023-07-10T17:14:53.679424+0000 osd.7 (osd.7) 66 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[65980]: cluster 2023-07-10T17:14:53.680938+0000 osd.7 (osd.7) 67 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[65980]: cluster 2023-07-10T17:14:53.682732+0000 osd.7 (osd.7) 68 : cluster 3 failed to encode map e114 with expected crc 2023-07-10T17:14:55.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:54 smithi116 bash[65980]: cluster 2023-07-10T17:14:54.665746+0000 mon.a (mon.0) 727 : cluster 0 osdmap e115: 8 total, 8 up, 8 in 2023-07-10T17:14:56.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:55 smithi192 bash[50362]: cluster 2023-07-10T17:14:54.843418+0000 osd.7 (osd.7) 69 : cluster 3 failed to encode map e115 with expected crc 2023-07-10T17:14:56.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:55 smithi116 bash[63900]: cluster 2023-07-10T17:14:54.843418+0000 osd.7 (osd.7) 69 : cluster 3 failed to encode map e115 with expected crc 2023-07-10T17:14:56.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:55 smithi116 bash[65980]: cluster 2023-07-10T17:14:54.843418+0000 osd.7 (osd.7) 69 : cluster 3 failed to encode map e115 with expected crc 2023-07-10T17:14:57.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:56 smithi192 bash[50362]: cluster 2023-07-10T17:14:55.506930+0000 mgr.y (mgr.25390) 293 : cluster 0 pgmap v210: 161 pgs: 1 peering, 7 active+clean+remapped, 28 active+undersized, 15 active+undersized+degraded, 110 active+clean; 604 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 104/999 objects degraded (10.410%) 2023-07-10T17:14:57.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:56 smithi192 bash[50362]: audit 2023-07-10T17:14:56.606829+0000 mon.a (mon.0) 728 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:57.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:56 smithi116 bash[63900]: cluster 2023-07-10T17:14:55.506930+0000 mgr.y (mgr.25390) 293 : cluster 0 pgmap v210: 161 pgs: 1 peering, 7 active+clean+remapped, 28 active+undersized, 15 active+undersized+degraded, 110 active+clean; 604 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 104/999 objects degraded (10.410%) 2023-07-10T17:14:57.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:56 smithi116 bash[63900]: audit 2023-07-10T17:14:56.606829+0000 mon.a (mon.0) 728 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:57.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:56 smithi116 bash[65980]: cluster 2023-07-10T17:14:55.506930+0000 mgr.y (mgr.25390) 293 : cluster 0 pgmap v210: 161 pgs: 1 peering, 7 active+clean+remapped, 28 active+undersized, 15 active+undersized+degraded, 110 active+clean; 604 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 104/999 objects degraded (10.410%) 2023-07-10T17:14:57.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:56 smithi116 bash[65980]: audit 2023-07-10T17:14:56.606829+0000 mon.a (mon.0) 728 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:14:58.092 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:57 smithi192 bash[50362]: cluster 2023-07-10T17:14:57.606281+0000 mon.a (mon.0) 729 : cluster 3 Health check update: Degraded data redundancy: 104/999 objects degraded (10.410%), 15 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:58.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:57 smithi116 bash[63900]: cluster 2023-07-10T17:14:57.606281+0000 mon.a (mon.0) 729 : cluster 3 Health check update: Degraded data redundancy: 104/999 objects degraded (10.410%), 15 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:58.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:57 smithi116 bash[65980]: cluster 2023-07-10T17:14:57.606281+0000 mon.a (mon.0) 729 : cluster 3 Health check update: Degraded data redundancy: 104/999 objects degraded (10.410%), 15 pgs degraded (PG_DEGRADED) 2023-07-10T17:14:58.973 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:58 smithi192 bash[50362]: cluster 2023-07-10T17:14:57.508575+0000 mgr.y (mgr.25390) 294 : cluster 0 pgmap v211: 161 pgs: 1 peering, 160 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:58.974 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:14:58 smithi192 bash[50362]: cluster 2023-07-10T17:14:57.691372+0000 mon.a (mon.0) 730 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 104/999 objects degraded (10.410%), 15 pgs degraded) 2023-07-10T17:14:59.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:58 smithi116 bash[63900]: cluster 2023-07-10T17:14:57.508575+0000 mgr.y (mgr.25390) 294 : cluster 0 pgmap v211: 161 pgs: 1 peering, 160 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:59.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:14:58 smithi116 bash[63900]: cluster 2023-07-10T17:14:57.691372+0000 mon.a (mon.0) 730 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 104/999 objects degraded (10.410%), 15 pgs degraded) 2023-07-10T17:14:59.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:58 smithi116 bash[65980]: cluster 2023-07-10T17:14:57.508575+0000 mgr.y (mgr.25390) 294 : cluster 0 pgmap v211: 161 pgs: 1 peering, 160 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:14:59.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:14:58 smithi116 bash[65980]: cluster 2023-07-10T17:14:57.691372+0000 mon.a (mon.0) 730 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 104/999 objects degraded (10.410%), 15 pgs degraded) 2023-07-10T17:15:01.055 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:00 smithi192 bash[50362]: cluster 2023-07-10T17:14:59.509553+0000 mgr.y (mgr.25390) 295 : cluster 0 pgmap v212: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:01.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:00 smithi116 bash[63900]: cluster 2023-07-10T17:14:59.509553+0000 mgr.y (mgr.25390) 295 : cluster 0 pgmap v212: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:01.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:00 smithi116 bash[65980]: cluster 2023-07-10T17:14:59.509553+0000 mgr.y (mgr.25390) 295 : cluster 0 pgmap v212: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:03.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:03 smithi192 bash[50362]: cluster 2023-07-10T17:15:01.510705+0000 mgr.y (mgr.25390) 296 : cluster 0 pgmap v213: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:03.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:03 smithi192 bash[50362]: audit 2023-07-10T17:15:02.092782+0000 mon.a (mon.0) 731 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:03.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:03 smithi192 bash[50362]: audit 2023-07-10T17:15:02.104183+0000 mon.a (mon.0) 732 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:03.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:03 smithi192 bash[50362]: audit 2023-07-10T17:15:02.106183+0000 mon.a (mon.0) 733 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:03.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:03 smithi192 bash[50362]: audit 2023-07-10T17:15:02.107631+0000 mon.a (mon.0) 734 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:15:03.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:03 smithi192 bash[50362]: cephadm 2023-07-10T17:15:02.109131+0000 mgr.y (mgr.25390) 297 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:15:03.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:03 smithi192 bash[50362]: cluster 2023-07-10T17:15:02.110426+0000 mgr.y (mgr.25390) 298 : cluster 0 pgmap v214: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:03.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:03 smithi192 bash[50362]: cluster 2023-07-10T17:15:02.110724+0000 mgr.y (mgr.25390) 299 : cluster 0 pgmap v215: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:03.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:03 smithi192 bash[50362]: audit 2023-07-10T17:15:02.118190+0000 mon.a (mon.0) 735 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:03.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:03 smithi192 bash[50362]: audit 2023-07-10T17:15:02.202833+0000 mon.a (mon.0) 736 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:15:03.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:03 smithi192 bash[50362]: audit 2023-07-10T17:15:02.209689+0000 mon.a (mon.0) 737 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:03.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:03 smithi192 bash[50362]: audit 2023-07-10T17:15:02.213692+0000 mon.a (mon.0) 738 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:03.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:03 smithi192 bash[50362]: audit 2023-07-10T17:15:02.215443+0000 mon.a (mon.0) 739 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:03.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:03 smithi192 bash[50362]: audit 2023-07-10T17:15:02.217615+0000 mon.a (mon.0) 740 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-07-10T17:15:03.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:03 smithi192 bash[50362]: audit 2023-07-10T17:15:02.217854+0000 mgr.y (mgr.25390) 300 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-07-10T17:15:03.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:03 smithi192 bash[50362]: cephadm 2023-07-10T17:15:02.219385+0000 mgr.y (mgr.25390) 301 : cephadm 1 Upgrade: osd.7 is safe to restart 2023-07-10T17:15:03.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[63900]: cluster 2023-07-10T17:15:01.510705+0000 mgr.y (mgr.25390) 296 : cluster 0 pgmap v213: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:03.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[63900]: audit 2023-07-10T17:15:02.092782+0000 mon.a (mon.0) 731 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:03.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[63900]: audit 2023-07-10T17:15:02.104183+0000 mon.a (mon.0) 732 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:03.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[63900]: audit 2023-07-10T17:15:02.106183+0000 mon.a (mon.0) 733 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:03.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[63900]: audit 2023-07-10T17:15:02.107631+0000 mon.a (mon.0) 734 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:15:03.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[63900]: cephadm 2023-07-10T17:15:02.109131+0000 mgr.y (mgr.25390) 297 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:15:03.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[63900]: cluster 2023-07-10T17:15:02.110426+0000 mgr.y (mgr.25390) 298 : cluster 0 pgmap v214: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:03.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[63900]: cluster 2023-07-10T17:15:02.110724+0000 mgr.y (mgr.25390) 299 : cluster 0 pgmap v215: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:03.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[63900]: audit 2023-07-10T17:15:02.118190+0000 mon.a (mon.0) 735 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:03.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[63900]: audit 2023-07-10T17:15:02.202833+0000 mon.a (mon.0) 736 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:15:03.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[63900]: audit 2023-07-10T17:15:02.209689+0000 mon.a (mon.0) 737 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:03.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[63900]: audit 2023-07-10T17:15:02.213692+0000 mon.a (mon.0) 738 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:03.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[63900]: audit 2023-07-10T17:15:02.215443+0000 mon.a (mon.0) 739 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:03.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[63900]: audit 2023-07-10T17:15:02.217615+0000 mon.a (mon.0) 740 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-07-10T17:15:03.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[63900]: audit 2023-07-10T17:15:02.217854+0000 mgr.y (mgr.25390) 300 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-07-10T17:15:03.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[63900]: cephadm 2023-07-10T17:15:02.219385+0000 mgr.y (mgr.25390) 301 : cephadm 1 Upgrade: osd.7 is safe to restart 2023-07-10T17:15:03.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[65980]: cluster 2023-07-10T17:15:01.510705+0000 mgr.y (mgr.25390) 296 : cluster 0 pgmap v213: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:03.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[65980]: audit 2023-07-10T17:15:02.092782+0000 mon.a (mon.0) 731 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:03.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[65980]: audit 2023-07-10T17:15:02.104183+0000 mon.a (mon.0) 732 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:03.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[65980]: audit 2023-07-10T17:15:02.106183+0000 mon.a (mon.0) 733 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:03.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[65980]: audit 2023-07-10T17:15:02.107631+0000 mon.a (mon.0) 734 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:15:03.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[65980]: cephadm 2023-07-10T17:15:02.109131+0000 mgr.y (mgr.25390) 297 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:15:03.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[65980]: cluster 2023-07-10T17:15:02.110426+0000 mgr.y (mgr.25390) 298 : cluster 0 pgmap v214: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:03.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[65980]: cluster 2023-07-10T17:15:02.110724+0000 mgr.y (mgr.25390) 299 : cluster 0 pgmap v215: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:03.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[65980]: audit 2023-07-10T17:15:02.118190+0000 mon.a (mon.0) 735 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:03.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[65980]: audit 2023-07-10T17:15:02.202833+0000 mon.a (mon.0) 736 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:15:03.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[65980]: audit 2023-07-10T17:15:02.209689+0000 mon.a (mon.0) 737 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:03.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[65980]: audit 2023-07-10T17:15:02.213692+0000 mon.a (mon.0) 738 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:03.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[65980]: audit 2023-07-10T17:15:02.215443+0000 mon.a (mon.0) 739 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:03.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[65980]: audit 2023-07-10T17:15:02.217615+0000 mon.a (mon.0) 740 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-07-10T17:15:03.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[65980]: audit 2023-07-10T17:15:02.217854+0000 mgr.y (mgr.25390) 300 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-07-10T17:15:03.399 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:03 smithi116 bash[65980]: cephadm 2023-07-10T17:15:02.219385+0000 mgr.y (mgr.25390) 301 : cephadm 1 Upgrade: osd.7 is safe to restart 2023-07-10T17:15:05.482 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:05 smithi192 bash[50362]: cluster 2023-07-10T17:15:04.111759+0000 mgr.y (mgr.25390) 302 : cluster 0 pgmap v216: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:05.490 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:05 smithi192 bash[50362]: audit 2023-07-10T17:15:04.399489+0000 mon.a (mon.0) 741 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:05.491 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:05 smithi192 bash[50362]: audit 2023-07-10T17:15:04.401005+0000 mon.a (mon.0) 742 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-10T17:15:05.491 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:05 smithi192 bash[50362]: audit 2023-07-10T17:15:04.402237+0000 mon.a (mon.0) 743 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:05.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:05 smithi116 bash[63900]: cluster 2023-07-10T17:15:04.111759+0000 mgr.y (mgr.25390) 302 : cluster 0 pgmap v216: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:05.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:05 smithi116 bash[63900]: audit 2023-07-10T17:15:04.399489+0000 mon.a (mon.0) 741 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:05.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:05 smithi116 bash[63900]: audit 2023-07-10T17:15:04.401005+0000 mon.a (mon.0) 742 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-10T17:15:05.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:05 smithi116 bash[63900]: audit 2023-07-10T17:15:04.402237+0000 mon.a (mon.0) 743 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:05.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:05 smithi116 bash[65980]: cluster 2023-07-10T17:15:04.111759+0000 mgr.y (mgr.25390) 302 : cluster 0 pgmap v216: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:05.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:05 smithi116 bash[65980]: audit 2023-07-10T17:15:04.399489+0000 mon.a (mon.0) 741 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:05.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:05 smithi116 bash[65980]: audit 2023-07-10T17:15:04.401005+0000 mon.a (mon.0) 742 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-10T17:15:05.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:05 smithi116 bash[65980]: audit 2023-07-10T17:15:04.402237+0000 mon.a (mon.0) 743 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:06.469 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:06 smithi192 bash[50362]: cephadm 2023-07-10T17:15:04.389532+0000 mgr.y (mgr.25390) 303 : cephadm 1 Upgrade: Updating osd.7 2023-07-10T17:15:06.470 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:06 smithi192 bash[50362]: cephadm 2023-07-10T17:15:04.405891+0000 mgr.y (mgr.25390) 304 : cephadm 1 Deploying daemon osd.7 on smithi192 2023-07-10T17:15:06.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:06 smithi116 bash[63900]: cephadm 2023-07-10T17:15:04.389532+0000 mgr.y (mgr.25390) 303 : cephadm 1 Upgrade: Updating osd.7 2023-07-10T17:15:06.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:06 smithi116 bash[63900]: cephadm 2023-07-10T17:15:04.405891+0000 mgr.y (mgr.25390) 304 : cephadm 1 Deploying daemon osd.7 on smithi192 2023-07-10T17:15:06.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:06 smithi116 bash[65980]: cephadm 2023-07-10T17:15:04.389532+0000 mgr.y (mgr.25390) 303 : cephadm 1 Upgrade: Updating osd.7 2023-07-10T17:15:06.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:06 smithi116 bash[65980]: cephadm 2023-07-10T17:15:04.405891+0000 mgr.y (mgr.25390) 304 : cephadm 1 Deploying daemon osd.7 on smithi192 2023-07-10T17:15:07.491 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:07 smithi192 bash[50362]: cluster 2023-07-10T17:15:06.113021+0000 mgr.y (mgr.25390) 305 : cluster 0 pgmap v217: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:07.491 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:07 smithi192 bash[50362]: audit 2023-07-10T17:15:06.639290+0000 mon.a (mon.0) 744 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:07.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:07 smithi116 bash[63900]: cluster 2023-07-10T17:15:06.113021+0000 mgr.y (mgr.25390) 305 : cluster 0 pgmap v217: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:07.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:07 smithi116 bash[63900]: audit 2023-07-10T17:15:06.639290+0000 mon.a (mon.0) 744 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:07.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:07 smithi116 bash[65980]: cluster 2023-07-10T17:15:06.113021+0000 mgr.y (mgr.25390) 305 : cluster 0 pgmap v217: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:07.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:07 smithi116 bash[65980]: audit 2023-07-10T17:15:06.639290+0000 mon.a (mon.0) 744 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:08.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:08 smithi192 bash[50362]: cluster 2023-07-10T17:15:08.114317+0000 mgr.y (mgr.25390) 306 : cluster 0 pgmap v218: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:08.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:08 smithi192 systemd[1]: Stopping Ceph osd.7 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:15:08.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:08 smithi116 bash[63900]: cluster 2023-07-10T17:15:08.114317+0000 mgr.y (mgr.25390) 306 : cluster 0 pgmap v218: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:08.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:08 smithi116 bash[65980]: cluster 2023-07-10T17:15:08.114317+0000 mgr.y (mgr.25390) 306 : cluster 0 pgmap v218: 161 pgs: 161 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:09.168 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:08 smithi192 bash[34832]: debug 2023-07-10T17:15:08.908+0000 7f2d47bcc700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:15:09.169 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:08 smithi192 bash[34832]: debug 2023-07-10T17:15:08.908+0000 7f2d47bcc700 -1 osd.7 115 *** Got signal Terminated *** 2023-07-10T17:15:09.169 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:08 smithi192 bash[34832]: debug 2023-07-10T17:15:08.908+0000 7f2d47bcc700 -1 osd.7 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-10T17:15:09.882 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[59010]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-osd.7 2023-07-10T17:15:09.883 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.168361+0000 mon.a (mon.0) 745 : cluster 0 osd.7 reported immediately failed by osd.4 2023-07-10T17:15:09.883 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.168511+0000 mon.a (mon.0) 746 : cluster 1 osd.7 failed (root=default,host=smithi192) (connection refused reported by osd.4) 2023-07-10T17:15:09.883 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.169014+0000 mon.a (mon.0) 747 : cluster 0 osd.7 reported immediately failed by osd.6 2023-07-10T17:15:09.883 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.169290+0000 mon.a (mon.0) 748 : cluster 0 osd.7 reported immediately failed by osd.3 2023-07-10T17:15:09.883 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.169517+0000 mon.a (mon.0) 749 : cluster 0 osd.7 reported immediately failed by osd.4 2023-07-10T17:15:09.883 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.169817+0000 mon.a (mon.0) 750 : cluster 0 osd.7 reported immediately failed by osd.0 2023-07-10T17:15:09.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.170114+0000 mon.a (mon.0) 751 : cluster 0 osd.7 reported immediately failed by osd.5 2023-07-10T17:15:09.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.170395+0000 mon.a (mon.0) 752 : cluster 0 osd.7 reported immediately failed by osd.3 2023-07-10T17:15:09.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.170695+0000 mon.a (mon.0) 753 : cluster 0 osd.7 reported immediately failed by osd.0 2023-07-10T17:15:09.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.171035+0000 mon.a (mon.0) 754 : cluster 0 osd.7 reported immediately failed by osd.6 2023-07-10T17:15:09.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.171340+0000 mon.a (mon.0) 755 : cluster 0 osd.7 reported immediately failed by osd.1 2023-07-10T17:15:09.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.171752+0000 mon.a (mon.0) 756 : cluster 0 osd.7 reported immediately failed by osd.5 2023-07-10T17:15:09.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.171968+0000 mon.a (mon.0) 757 : cluster 0 osd.7 reported immediately failed by osd.2 2023-07-10T17:15:09.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.172358+0000 mon.a (mon.0) 758 : cluster 0 osd.7 reported immediately failed by osd.1 2023-07-10T17:15:09.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.172553+0000 mon.a (mon.0) 759 : cluster 0 osd.7 reported immediately failed by osd.2 2023-07-10T17:15:09.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.571832+0000 mon.a (mon.0) 760 : cluster 0 osd.7 reported immediately failed by osd.4 2023-07-10T17:15:09.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.572719+0000 mon.a (mon.0) 761 : cluster 0 osd.7 reported immediately failed by osd.4 2023-07-10T17:15:09.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.573249+0000 mon.a (mon.0) 762 : cluster 0 osd.7 reported immediately failed by osd.6 2023-07-10T17:15:09.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.573598+0000 mon.a (mon.0) 763 : cluster 0 osd.7 reported immediately failed by osd.3 2023-07-10T17:15:09.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.573917+0000 mon.a (mon.0) 764 : cluster 0 osd.7 reported immediately failed by osd.1 2023-07-10T17:15:09.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.574164+0000 mon.a (mon.0) 765 : cluster 0 osd.7 reported immediately failed by osd.6 2023-07-10T17:15:09.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.574465+0000 mon.a (mon.0) 766 : cluster 0 osd.7 reported immediately failed by osd.3 2023-07-10T17:15:09.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.574678+0000 mon.a (mon.0) 767 : cluster 0 osd.7 reported immediately failed by osd.2 2023-07-10T17:15:09.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.575157+0000 mon.a (mon.0) 768 : cluster 0 osd.7 reported immediately failed by osd.5 2023-07-10T17:15:09.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.575517+0000 mon.a (mon.0) 769 : cluster 0 osd.7 reported immediately failed by osd.0 2023-07-10T17:15:09.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.575917+0000 mon.a (mon.0) 770 : cluster 0 osd.7 reported immediately failed by osd.5 2023-07-10T17:15:09.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.576294+0000 mon.a (mon.0) 771 : cluster 0 osd.7 reported immediately failed by osd.1 2023-07-10T17:15:09.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.576715+0000 mon.a (mon.0) 772 : cluster 0 osd.7 reported immediately failed by osd.2 2023-07-10T17:15:09.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:09 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.577135+0000 mon.a (mon.0) 773 : cluster 0 osd.7 reported immediately failed by osd.0 2023-07-10T17:15:09.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.168361+0000 mon.a (mon.0) 745 : cluster 0 osd.7 reported immediately failed by osd.4 2023-07-10T17:15:09.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.168511+0000 mon.a (mon.0) 746 : cluster 1 osd.7 failed (root=default,host=smithi192) (connection refused reported by osd.4) 2023-07-10T17:15:09.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.169014+0000 mon.a (mon.0) 747 : cluster 0 osd.7 reported immediately failed by osd.6 2023-07-10T17:15:09.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.169290+0000 mon.a (mon.0) 748 : cluster 0 osd.7 reported immediately failed by osd.3 2023-07-10T17:15:09.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.169517+0000 mon.a (mon.0) 749 : cluster 0 osd.7 reported immediately failed by osd.4 2023-07-10T17:15:09.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.169817+0000 mon.a (mon.0) 750 : cluster 0 osd.7 reported immediately failed by osd.0 2023-07-10T17:15:09.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.170114+0000 mon.a (mon.0) 751 : cluster 0 osd.7 reported immediately failed by osd.5 2023-07-10T17:15:09.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.170395+0000 mon.a (mon.0) 752 : cluster 0 osd.7 reported immediately failed by osd.3 2023-07-10T17:15:09.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.170695+0000 mon.a (mon.0) 753 : cluster 0 osd.7 reported immediately failed by osd.0 2023-07-10T17:15:09.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.171035+0000 mon.a (mon.0) 754 : cluster 0 osd.7 reported immediately failed by osd.6 2023-07-10T17:15:09.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.171340+0000 mon.a (mon.0) 755 : cluster 0 osd.7 reported immediately failed by osd.1 2023-07-10T17:15:09.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.171752+0000 mon.a (mon.0) 756 : cluster 0 osd.7 reported immediately failed by osd.5 2023-07-10T17:15:09.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.171968+0000 mon.a (mon.0) 757 : cluster 0 osd.7 reported immediately failed by osd.2 2023-07-10T17:15:09.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.172358+0000 mon.a (mon.0) 758 : cluster 0 osd.7 reported immediately failed by osd.1 2023-07-10T17:15:09.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.172553+0000 mon.a (mon.0) 759 : cluster 0 osd.7 reported immediately failed by osd.2 2023-07-10T17:15:09.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.571832+0000 mon.a (mon.0) 760 : cluster 0 osd.7 reported immediately failed by osd.4 2023-07-10T17:15:09.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.572719+0000 mon.a (mon.0) 761 : cluster 0 osd.7 reported immediately failed by osd.4 2023-07-10T17:15:09.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.573249+0000 mon.a (mon.0) 762 : cluster 0 osd.7 reported immediately failed by osd.6 2023-07-10T17:15:09.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.573598+0000 mon.a (mon.0) 763 : cluster 0 osd.7 reported immediately failed by osd.3 2023-07-10T17:15:09.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.573917+0000 mon.a (mon.0) 764 : cluster 0 osd.7 reported immediately failed by osd.1 2023-07-10T17:15:09.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.574164+0000 mon.a (mon.0) 765 : cluster 0 osd.7 reported immediately failed by osd.6 2023-07-10T17:15:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.574465+0000 mon.a (mon.0) 766 : cluster 0 osd.7 reported immediately failed by osd.3 2023-07-10T17:15:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.574678+0000 mon.a (mon.0) 767 : cluster 0 osd.7 reported immediately failed by osd.2 2023-07-10T17:15:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.575157+0000 mon.a (mon.0) 768 : cluster 0 osd.7 reported immediately failed by osd.5 2023-07-10T17:15:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.575517+0000 mon.a (mon.0) 769 : cluster 0 osd.7 reported immediately failed by osd.0 2023-07-10T17:15:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.575917+0000 mon.a (mon.0) 770 : cluster 0 osd.7 reported immediately failed by osd.5 2023-07-10T17:15:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.576294+0000 mon.a (mon.0) 771 : cluster 0 osd.7 reported immediately failed by osd.1 2023-07-10T17:15:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.576715+0000 mon.a (mon.0) 772 : cluster 0 osd.7 reported immediately failed by osd.2 2023-07-10T17:15:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.577135+0000 mon.a (mon.0) 773 : cluster 0 osd.7 reported immediately failed by osd.0 2023-07-10T17:15:09.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.168361+0000 mon.a (mon.0) 745 : cluster 0 osd.7 reported immediately failed by osd.4 2023-07-10T17:15:09.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.168511+0000 mon.a (mon.0) 746 : cluster 1 osd.7 failed (root=default,host=smithi192) (connection refused reported by osd.4) 2023-07-10T17:15:09.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.169014+0000 mon.a (mon.0) 747 : cluster 0 osd.7 reported immediately failed by osd.6 2023-07-10T17:15:09.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.169290+0000 mon.a (mon.0) 748 : cluster 0 osd.7 reported immediately failed by osd.3 2023-07-10T17:15:09.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.169517+0000 mon.a (mon.0) 749 : cluster 0 osd.7 reported immediately failed by osd.4 2023-07-10T17:15:09.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.169817+0000 mon.a (mon.0) 750 : cluster 0 osd.7 reported immediately failed by osd.0 2023-07-10T17:15:09.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.170114+0000 mon.a (mon.0) 751 : cluster 0 osd.7 reported immediately failed by osd.5 2023-07-10T17:15:09.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.170395+0000 mon.a (mon.0) 752 : cluster 0 osd.7 reported immediately failed by osd.3 2023-07-10T17:15:09.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.170695+0000 mon.a (mon.0) 753 : cluster 0 osd.7 reported immediately failed by osd.0 2023-07-10T17:15:09.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.171035+0000 mon.a (mon.0) 754 : cluster 0 osd.7 reported immediately failed by osd.6 2023-07-10T17:15:09.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.171340+0000 mon.a (mon.0) 755 : cluster 0 osd.7 reported immediately failed by osd.1 2023-07-10T17:15:09.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.171752+0000 mon.a (mon.0) 756 : cluster 0 osd.7 reported immediately failed by osd.5 2023-07-10T17:15:09.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.171968+0000 mon.a (mon.0) 757 : cluster 0 osd.7 reported immediately failed by osd.2 2023-07-10T17:15:09.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.172358+0000 mon.a (mon.0) 758 : cluster 0 osd.7 reported immediately failed by osd.1 2023-07-10T17:15:09.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.172553+0000 mon.a (mon.0) 759 : cluster 0 osd.7 reported immediately failed by osd.2 2023-07-10T17:15:09.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.571832+0000 mon.a (mon.0) 760 : cluster 0 osd.7 reported immediately failed by osd.4 2023-07-10T17:15:09.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.572719+0000 mon.a (mon.0) 761 : cluster 0 osd.7 reported immediately failed by osd.4 2023-07-10T17:15:09.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.573249+0000 mon.a (mon.0) 762 : cluster 0 osd.7 reported immediately failed by osd.6 2023-07-10T17:15:09.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.573598+0000 mon.a (mon.0) 763 : cluster 0 osd.7 reported immediately failed by osd.3 2023-07-10T17:15:09.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.573917+0000 mon.a (mon.0) 764 : cluster 0 osd.7 reported immediately failed by osd.1 2023-07-10T17:15:09.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.574164+0000 mon.a (mon.0) 765 : cluster 0 osd.7 reported immediately failed by osd.6 2023-07-10T17:15:09.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.574465+0000 mon.a (mon.0) 766 : cluster 0 osd.7 reported immediately failed by osd.3 2023-07-10T17:15:09.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.574678+0000 mon.a (mon.0) 767 : cluster 0 osd.7 reported immediately failed by osd.2 2023-07-10T17:15:09.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.575157+0000 mon.a (mon.0) 768 : cluster 0 osd.7 reported immediately failed by osd.5 2023-07-10T17:15:09.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.575517+0000 mon.a (mon.0) 769 : cluster 0 osd.7 reported immediately failed by osd.0 2023-07-10T17:15:09.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.575917+0000 mon.a (mon.0) 770 : cluster 0 osd.7 reported immediately failed by osd.5 2023-07-10T17:15:09.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.576294+0000 mon.a (mon.0) 771 : cluster 0 osd.7 reported immediately failed by osd.1 2023-07-10T17:15:09.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.576715+0000 mon.a (mon.0) 772 : cluster 0 osd.7 reported immediately failed by osd.2 2023-07-10T17:15:09.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:09 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.577135+0000 mon.a (mon.0) 773 : cluster 0 osd.7 reported immediately failed by osd.0 2023-07-10T17:15:10.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:10 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.623266+0000 mon.a (mon.0) 774 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:15:10.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:10 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.623347+0000 mon.a (mon.0) 775 : cluster 3 Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2023-07-10T17:15:10.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:10 smithi192 bash[50362]: cluster 2023-07-10T17:15:09.639888+0000 mon.a (mon.0) 776 : cluster 0 osdmap e116: 8 total, 7 up, 8 in 2023-07-10T17:15:10.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:10 smithi192 bash[50362]: cluster 2023-07-10T17:15:10.115048+0000 mgr.y (mgr.25390) 307 : cluster 0 pgmap v220: 161 pgs: 23 stale+active+clean, 138 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:10.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:10 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.623266+0000 mon.a (mon.0) 774 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:15:10.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:10 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.623347+0000 mon.a (mon.0) 775 : cluster 3 Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2023-07-10T17:15:10.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:10 smithi116 bash[63900]: cluster 2023-07-10T17:15:09.639888+0000 mon.a (mon.0) 776 : cluster 0 osdmap e116: 8 total, 7 up, 8 in 2023-07-10T17:15:10.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:10 smithi116 bash[63900]: cluster 2023-07-10T17:15:10.115048+0000 mgr.y (mgr.25390) 307 : cluster 0 pgmap v220: 161 pgs: 23 stale+active+clean, 138 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:10.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:10 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.623266+0000 mon.a (mon.0) 774 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-07-10T17:15:10.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:10 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.623347+0000 mon.a (mon.0) 775 : cluster 3 Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2023-07-10T17:15:10.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:10 smithi116 bash[65980]: cluster 2023-07-10T17:15:09.639888+0000 mon.a (mon.0) 776 : cluster 0 osdmap e116: 8 total, 7 up, 8 in 2023-07-10T17:15:10.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:10 smithi116 bash[65980]: cluster 2023-07-10T17:15:10.115048+0000 mgr.y (mgr.25390) 307 : cluster 0 pgmap v220: 161 pgs: 23 stale+active+clean, 138 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:11.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:11 smithi192 bash[50362]: cluster 2023-07-10T17:15:10.657664+0000 mon.a (mon.0) 777 : cluster 0 osdmap e117: 8 total, 7 up, 8 in 2023-07-10T17:15:12.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:11 smithi116 bash[63900]: cluster 2023-07-10T17:15:10.657664+0000 mon.a (mon.0) 777 : cluster 0 osdmap e117: 8 total, 7 up, 8 in 2023-07-10T17:15:12.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:11 smithi116 bash[65980]: cluster 2023-07-10T17:15:10.657664+0000 mon.a (mon.0) 777 : cluster 0 osdmap e117: 8 total, 7 up, 8 in 2023-07-10T17:15:12.207 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:11 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.7.service: Succeeded. 2023-07-10T17:15:12.208 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:11 smithi192 systemd[1]: Stopped Ceph osd.7 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:15:12.638 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:12 smithi192 systemd[1]: Started Ceph osd.7 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:15:13.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:13 smithi192 bash[50362]: cluster 2023-07-10T17:15:12.116694+0000 mgr.y (mgr.25390) 308 : cluster 0 pgmap v222: 161 pgs: 33 active+undersized, 11 active+undersized+degraded, 4 peering, 10 stale+active+clean, 103 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 47/999 objects degraded (4.705%) 2023-07-10T17:15:13.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:13 smithi192 bash[50362]: audit 2023-07-10T17:15:12.248641+0000 mon.a (mon.0) 778 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:13.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:13 smithi192 bash[50362]: audit 2023-07-10T17:15:12.258645+0000 mon.a (mon.0) 779 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:13.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:13 smithi192 bash[50362]: cluster 2023-07-10T17:15:12.647226+0000 mon.a (mon.0) 780 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2023-07-10T17:15:13.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:13 smithi192 bash[50362]: cluster 2023-07-10T17:15:12.647288+0000 mon.a (mon.0) 781 : cluster 3 Health check failed: Degraded data redundancy: 47/999 objects degraded (4.705%), 11 pgs degraded (PG_DEGRADED) 2023-07-10T17:15:13.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:13 smithi192 bash[50362]: audit 2023-07-10T17:15:12.663612+0000 mon.a (mon.0) 782 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:13.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:13 smithi192 bash[50362]: audit 2023-07-10T17:15:12.676546+0000 mon.a (mon.0) 783 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:13.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:13 smithi116 bash[63900]: cluster 2023-07-10T17:15:12.116694+0000 mgr.y (mgr.25390) 308 : cluster 0 pgmap v222: 161 pgs: 33 active+undersized, 11 active+undersized+degraded, 4 peering, 10 stale+active+clean, 103 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 47/999 objects degraded (4.705%) 2023-07-10T17:15:13.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:13 smithi116 bash[63900]: audit 2023-07-10T17:15:12.248641+0000 mon.a (mon.0) 778 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:13.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:13 smithi116 bash[63900]: audit 2023-07-10T17:15:12.258645+0000 mon.a (mon.0) 779 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:13.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:13 smithi116 bash[63900]: cluster 2023-07-10T17:15:12.647226+0000 mon.a (mon.0) 780 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2023-07-10T17:15:13.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:13 smithi116 bash[63900]: cluster 2023-07-10T17:15:12.647288+0000 mon.a (mon.0) 781 : cluster 3 Health check failed: Degraded data redundancy: 47/999 objects degraded (4.705%), 11 pgs degraded (PG_DEGRADED) 2023-07-10T17:15:13.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:13 smithi116 bash[63900]: audit 2023-07-10T17:15:12.663612+0000 mon.a (mon.0) 782 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:13.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:13 smithi116 bash[63900]: audit 2023-07-10T17:15:12.676546+0000 mon.a (mon.0) 783 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:13.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:13 smithi116 bash[65980]: cluster 2023-07-10T17:15:12.116694+0000 mgr.y (mgr.25390) 308 : cluster 0 pgmap v222: 161 pgs: 33 active+undersized, 11 active+undersized+degraded, 4 peering, 10 stale+active+clean, 103 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 47/999 objects degraded (4.705%) 2023-07-10T17:15:13.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:13 smithi116 bash[65980]: audit 2023-07-10T17:15:12.248641+0000 mon.a (mon.0) 778 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:13.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:13 smithi116 bash[65980]: audit 2023-07-10T17:15:12.258645+0000 mon.a (mon.0) 779 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:13.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:13 smithi116 bash[65980]: cluster 2023-07-10T17:15:12.647226+0000 mon.a (mon.0) 780 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2023-07-10T17:15:13.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:13 smithi116 bash[65980]: cluster 2023-07-10T17:15:12.647288+0000 mon.a (mon.0) 781 : cluster 3 Health check failed: Degraded data redundancy: 47/999 objects degraded (4.705%), 11 pgs degraded (PG_DEGRADED) 2023-07-10T17:15:13.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:13 smithi116 bash[65980]: audit 2023-07-10T17:15:12.663612+0000 mon.a (mon.0) 782 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:13.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:13 smithi116 bash[65980]: audit 2023-07-10T17:15:12.676546+0000 mon.a (mon.0) 783 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:14.137 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:13 smithi192 bash[59288]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-07-10T17:15:14.137 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:13 smithi192 bash[59288]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2023-07-10T17:15:14.137 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:13 smithi192 bash[59288]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2023-07-10T17:15:14.138 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:13 smithi192 bash[59288]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2023-07-10T17:15:14.138 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:13 smithi192 bash[59288]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2023-07-10T17:15:14.138 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:13 smithi192 bash[59288]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-07-10T17:15:14.138 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:13 smithi192 bash[59288]: --> ceph-volume raw activate successful for osd ID: 7 2023-07-10T17:15:15.534 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:15 smithi192 bash[50362]: cluster 2023-07-10T17:15:14.117954+0000 mgr.y (mgr.25390) 309 : cluster 0 pgmap v223: 161 pgs: 38 active+undersized, 16 active+undersized+degraded, 4 peering, 7 stale+active+clean, 96 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 73/999 objects degraded (7.307%) 2023-07-10T17:15:15.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:15 smithi116 bash[63900]: cluster 2023-07-10T17:15:14.117954+0000 mgr.y (mgr.25390) 309 : cluster 0 pgmap v223: 161 pgs: 38 active+undersized, 16 active+undersized+degraded, 4 peering, 7 stale+active+clean, 96 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 73/999 objects degraded (7.307%) 2023-07-10T17:15:15.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:15 smithi116 bash[65980]: cluster 2023-07-10T17:15:14.117954+0000 mgr.y (mgr.25390) 309 : cluster 0 pgmap v223: 161 pgs: 38 active+undersized, 16 active+undersized+degraded, 4 peering, 7 stale+active+clean, 96 active+clean; 604 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 73/999 objects degraded (7.307%) 2023-07-10T17:15:15.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:15 smithi192 bash[59572]: debug 2023-07-10T17:15:15.528+0000 7f3f8800a700 1 -- 172.21.15.192:0/2131999480 <== mon.1 v2:172.21.15.116:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558faebe44e0 con 0x558faec3ec00 2023-07-10T17:15:16.387 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:16 smithi192 bash[59572]: debug 2023-07-10T17:15:16.068+0000 7f3f9118f640 -1 Falling back to public interface 2023-07-10T17:15:16.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:16 smithi192 bash[50362]: cluster 2023-07-10T17:15:16.119591+0000 mgr.y (mgr.25390) 310 : cluster 0 pgmap v224: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 132/999 objects degraded (13.213%) 2023-07-10T17:15:16.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:16 smithi192 bash[50362]: cluster 2023-07-10T17:15:16.251953+0000 mon.a (mon.0) 784 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2023-07-10T17:15:16.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:16 smithi116 bash[63900]: cluster 2023-07-10T17:15:16.119591+0000 mgr.y (mgr.25390) 310 : cluster 0 pgmap v224: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 132/999 objects degraded (13.213%) 2023-07-10T17:15:16.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:16 smithi116 bash[63900]: cluster 2023-07-10T17:15:16.251953+0000 mon.a (mon.0) 784 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2023-07-10T17:15:16.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:16 smithi116 bash[65980]: cluster 2023-07-10T17:15:16.119591+0000 mgr.y (mgr.25390) 310 : cluster 0 pgmap v224: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 132/999 objects degraded (13.213%) 2023-07-10T17:15:16.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:16 smithi116 bash[65980]: cluster 2023-07-10T17:15:16.251953+0000 mon.a (mon.0) 784 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2023-07-10T17:15:18.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:18 smithi192 bash[50362]: cluster 2023-07-10T17:15:18.121121+0000 mgr.y (mgr.25390) 311 : cluster 0 pgmap v225: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 132/999 objects degraded (13.213%) 2023-07-10T17:15:18.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:18 smithi116 bash[63900]: cluster 2023-07-10T17:15:18.121121+0000 mgr.y (mgr.25390) 311 : cluster 0 pgmap v225: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 132/999 objects degraded (13.213%) 2023-07-10T17:15:18.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:18 smithi116 bash[65980]: cluster 2023-07-10T17:15:18.121121+0000 mgr.y (mgr.25390) 311 : cluster 0 pgmap v225: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 132/999 objects degraded (13.213%) 2023-07-10T17:15:19.637 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:19 smithi192 bash[59572]: debug 2023-07-10T17:15:19.176+0000 7f3f9118f640 -1 osd.7 115 log_to_monitors true 2023-07-10T17:15:20.137 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:15:19 smithi192 bash[59572]: debug 2023-07-10T17:15:19.764+0000 7f3f80df9700 -1 osd.7 115 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-10T17:15:20.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:19 smithi192 bash[50362]: audit 2023-07-10T17:15:18.685107+0000 mon.a (mon.0) 785 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:20.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:19 smithi192 bash[50362]: audit 2023-07-10T17:15:18.692401+0000 mon.a (mon.0) 786 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:20.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:19 smithi192 bash[50362]: audit 2023-07-10T17:15:19.185198+0000 mon.a (mon.0) 787 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-10T17:15:20.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:19 smithi192 bash[50362]: audit 2023-07-10T17:15:19.185647+0000 mon.b (mon.2) 7 : audit 1 from='osd.7 [v2:172.21.15.192:6824/1537181775,v1:172.21.15.192:6825/1537181775]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-10T17:15:20.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:19 smithi192 bash[50362]: audit 2023-07-10T17:15:19.491802+0000 mon.a (mon.0) 788 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:20.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:19 smithi192 bash[50362]: audit 2023-07-10T17:15:19.503746+0000 mon.a (mon.0) 789 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:20.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:19 smithi116 bash[63900]: audit 2023-07-10T17:15:18.685107+0000 mon.a (mon.0) 785 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:20.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:19 smithi116 bash[63900]: audit 2023-07-10T17:15:18.692401+0000 mon.a (mon.0) 786 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:20.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:19 smithi116 bash[63900]: audit 2023-07-10T17:15:19.185198+0000 mon.a (mon.0) 787 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-10T17:15:20.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:19 smithi116 bash[63900]: audit 2023-07-10T17:15:19.185647+0000 mon.b (mon.2) 7 : audit 1 from='osd.7 [v2:172.21.15.192:6824/1537181775,v1:172.21.15.192:6825/1537181775]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-10T17:15:20.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:19 smithi116 bash[63900]: audit 2023-07-10T17:15:19.491802+0000 mon.a (mon.0) 788 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:20.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:19 smithi116 bash[63900]: audit 2023-07-10T17:15:19.503746+0000 mon.a (mon.0) 789 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:20.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:19 smithi116 bash[65980]: audit 2023-07-10T17:15:18.685107+0000 mon.a (mon.0) 785 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:20.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:19 smithi116 bash[65980]: audit 2023-07-10T17:15:18.692401+0000 mon.a (mon.0) 786 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:20.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:19 smithi116 bash[65980]: audit 2023-07-10T17:15:19.185198+0000 mon.a (mon.0) 787 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-10T17:15:20.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:19 smithi116 bash[65980]: audit 2023-07-10T17:15:19.185647+0000 mon.b (mon.2) 7 : audit 1 from='osd.7 [v2:172.21.15.192:6824/1537181775,v1:172.21.15.192:6825/1537181775]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-10T17:15:20.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:19 smithi116 bash[65980]: audit 2023-07-10T17:15:19.491802+0000 mon.a (mon.0) 788 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:20.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:19 smithi116 bash[65980]: audit 2023-07-10T17:15:19.503746+0000 mon.a (mon.0) 789 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:20.968 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:15:21.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:20 smithi192 bash[50362]: audit 2023-07-10T17:15:19.698199+0000 mon.a (mon.0) 790 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-10T17:15:21.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:20 smithi192 bash[50362]: cluster 2023-07-10T17:15:19.698297+0000 mon.a (mon.0) 791 : cluster 0 osdmap e118: 8 total, 7 up, 8 in 2023-07-10T17:15:21.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:20 smithi192 bash[50362]: audit 2023-07-10T17:15:19.702291+0000 mon.a (mon.0) 792 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:15:21.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:20 smithi192 bash[50362]: audit 2023-07-10T17:15:19.702793+0000 mon.b (mon.2) 8 : audit 1 from='osd.7 [v2:172.21.15.192:6824/1537181775,v1:172.21.15.192:6825/1537181775]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:15:21.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:20 smithi192 bash[50362]: cluster 2023-07-10T17:15:20.122247+0000 mgr.y (mgr.25390) 312 : cluster 0 pgmap v227: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 132/999 objects degraded (13.213%) 2023-07-10T17:15:21.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:20 smithi116 bash[65980]: audit 2023-07-10T17:15:19.698199+0000 mon.a (mon.0) 790 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-10T17:15:21.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:20 smithi116 bash[65980]: cluster 2023-07-10T17:15:19.698297+0000 mon.a (mon.0) 791 : cluster 0 osdmap e118: 8 total, 7 up, 8 in 2023-07-10T17:15:21.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:20 smithi116 bash[65980]: audit 2023-07-10T17:15:19.702291+0000 mon.a (mon.0) 792 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:15:21.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:20 smithi116 bash[65980]: audit 2023-07-10T17:15:19.702793+0000 mon.b (mon.2) 8 : audit 1 from='osd.7 [v2:172.21.15.192:6824/1537181775,v1:172.21.15.192:6825/1537181775]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:15:21.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:20 smithi116 bash[65980]: cluster 2023-07-10T17:15:20.122247+0000 mgr.y (mgr.25390) 312 : cluster 0 pgmap v227: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 132/999 objects degraded (13.213%) 2023-07-10T17:15:21.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:20 smithi116 bash[63900]: audit 2023-07-10T17:15:19.698199+0000 mon.a (mon.0) 790 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-10T17:15:21.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:20 smithi116 bash[63900]: cluster 2023-07-10T17:15:19.698297+0000 mon.a (mon.0) 791 : cluster 0 osdmap e118: 8 total, 7 up, 8 in 2023-07-10T17:15:21.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:20 smithi116 bash[63900]: audit 2023-07-10T17:15:19.702291+0000 mon.a (mon.0) 792 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:15:21.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:20 smithi116 bash[63900]: audit 2023-07-10T17:15:19.702793+0000 mon.b (mon.2) 8 : audit 1 from='osd.7 [v2:172.21.15.192:6824/1537181775,v1:172.21.15.192:6825/1537181775]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-07-10T17:15:21.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:20 smithi116 bash[63900]: cluster 2023-07-10T17:15:20.122247+0000 mgr.y (mgr.25390) 312 : cluster 0 pgmap v227: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 132/999 objects degraded (13.213%) 2023-07-10T17:15:21.705 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:15:21.706 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (8m) 116s ago 14m 10.9M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:15:21.706 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (7m) 3s ago 14m 26.0M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:15:21.706 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 error 116s ago 12m - - 2023-07-10T17:15:21.706 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (6m) 3s ago 19m 430M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:15:21.707 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (7m) 116s ago 20m 475M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:15:21.707 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (5m) 116s ago 21m 59.6M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:15:21.707 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (4m) 3s ago 19m 55.4M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 74a082a2e795 2023-07-10T17:15:21.707 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (5m) 116s ago 19m 49.9M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:15:21.707 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (8m) 116s ago 14m 9371k - 0.18.1 e5a616e4b9cf 3bd058d91353 2023-07-10T17:15:21.707 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (8m) 3s ago 14m 3944k - 0.18.1 e5a616e4b9cf 63551615c679 2023-07-10T17:15:21.707 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (3m) 116s ago 18m 53.2M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 0562f587fbca 2023-07-10T17:15:21.708 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (3m) 116s ago 18m 51.6M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 5b2a12a0bfc9 2023-07-10T17:15:21.708 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (2m) 116s ago 17m 47.9M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 88096bc0faa9 2023-07-10T17:15:21.708 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (2m) 116s ago 17m - 4096M 18.1.2-30-g8507faa7 17e8f0d79247 61ae280c471b 2023-07-10T17:15:21.708 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (92s) 3s ago 17m 54.3M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 44d809c18118 2023-07-10T17:15:21.708 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (64s) 3s ago 16m 50.2M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 179270eb7044 2023-07-10T17:15:21.708 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (35s) 3s ago 16m 50.5M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 b680bfbe9a0c 2023-07-10T17:15:21.708 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (7s) 3s ago 15m - 4096M 18.1.2-30-g8507faa7 17e8f0d79247 bd3788803e68 2023-07-10T17:15:21.709 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 error 3s ago 15m - - 2023-07-10T17:15:21.709 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (12m) 116s ago 12m 55.9M - 16.2.0 24ecd6d5f14c dd4c57bec826 2023-07-10T17:15:21.709 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (12m) 3s ago 12m 56.4M - 16.2.0 24ecd6d5f14c 01551ddefdcb 2023-07-10T17:15:21.709 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (12m) 116s ago 12m 55.5M - 16.2.0 24ecd6d5f14c 9ad1cc6126b4 2023-07-10T17:15:21.709 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (12m) 3s ago 12m 58.0M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:15:22.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:21 smithi192 bash[50362]: cluster 2023-07-10T17:15:20.695371+0000 mon.a (mon.0) 793 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:15:22.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:21 smithi192 bash[50362]: cluster 2023-07-10T17:15:20.711643+0000 mon.a (mon.0) 794 : cluster 1 osd.7 [v2:172.21.15.192:6824/1537181775,v1:172.21.15.192:6825/1537181775] boot 2023-07-10T17:15:22.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:21 smithi192 bash[50362]: cluster 2023-07-10T17:15:20.711877+0000 mon.a (mon.0) 795 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2023-07-10T17:15:22.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:21 smithi192 bash[50362]: audit 2023-07-10T17:15:20.713047+0000 mon.a (mon.0) 796 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T17:15:22.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:21 smithi192 bash[50362]: audit 2023-07-10T17:15:20.948726+0000 mgr.y (mgr.25390) 313 : audit 0 from='client.34713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:22.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:21 smithi192 bash[50362]: audit 2023-07-10T17:15:21.672660+0000 mon.a (mon.0) 797 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:22.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:21 smithi116 bash[65980]: cluster 2023-07-10T17:15:20.695371+0000 mon.a (mon.0) 793 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:15:22.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:21 smithi116 bash[65980]: cluster 2023-07-10T17:15:20.711643+0000 mon.a (mon.0) 794 : cluster 1 osd.7 [v2:172.21.15.192:6824/1537181775,v1:172.21.15.192:6825/1537181775] boot 2023-07-10T17:15:22.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:21 smithi116 bash[65980]: cluster 2023-07-10T17:15:20.711877+0000 mon.a (mon.0) 795 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2023-07-10T17:15:22.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:21 smithi116 bash[65980]: audit 2023-07-10T17:15:20.713047+0000 mon.a (mon.0) 796 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T17:15:22.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:21 smithi116 bash[65980]: audit 2023-07-10T17:15:20.948726+0000 mgr.y (mgr.25390) 313 : audit 0 from='client.34713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:22.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:21 smithi116 bash[65980]: audit 2023-07-10T17:15:21.672660+0000 mon.a (mon.0) 797 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:22.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:21 smithi116 bash[63900]: cluster 2023-07-10T17:15:20.695371+0000 mon.a (mon.0) 793 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-07-10T17:15:22.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:21 smithi116 bash[63900]: cluster 2023-07-10T17:15:20.711643+0000 mon.a (mon.0) 794 : cluster 1 osd.7 [v2:172.21.15.192:6824/1537181775,v1:172.21.15.192:6825/1537181775] boot 2023-07-10T17:15:22.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:21 smithi116 bash[63900]: cluster 2023-07-10T17:15:20.711877+0000 mon.a (mon.0) 795 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2023-07-10T17:15:22.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:21 smithi116 bash[63900]: audit 2023-07-10T17:15:20.713047+0000 mon.a (mon.0) 796 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-10T17:15:22.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:21 smithi116 bash[63900]: audit 2023-07-10T17:15:20.948726+0000 mgr.y (mgr.25390) 313 : audit 0 from='client.34713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:22.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:21 smithi116 bash[63900]: audit 2023-07-10T17:15:21.672660+0000 mon.a (mon.0) 797 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:22.151 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:15:22.151 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:15:22.151 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:15:22.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:15:22.152 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:15:22.152 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:15:22.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:15:22.152 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:15:22.152 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 8 2023-07-10T17:15:22.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:15:22.153 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:15:22.153 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2023-07-10T17:15:22.153 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:15:22.153 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:15:22.153 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4, 2023-07-10T17:15:22.153 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 13 2023-07-10T17:15:22.153 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:15:22.154 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:15:22.514 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:15:22.515 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:15:22.515 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:15:22.515 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:15:22.516 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:15:22.516 INFO:teuthology.orchestra.run.smithi116.stdout: "osd", 2023-07-10T17:15:22.516 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2023-07-10T17:15:22.516 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2023-07-10T17:15:22.517 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:15:22.517 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "13/23 daemons upgraded", 2023-07-10T17:15:22.517 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading osd daemons", 2023-07-10T17:15:22.517 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:15:22.518 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:15:22.943 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 2 failed cephadm daemon(s); all OSDs are running quincy or later but require_osd_release < quincy; Degraded data redundancy: 98/999 objects degraded (9.810%), 18 pgs degraded 2023-07-10T17:15:22.943 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:15:22.943 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:15:22.943 INFO:teuthology.orchestra.run.smithi116.stdout: daemon prometheus.a on smithi192 is in error state 2023-07-10T17:15:22.943 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running quincy or later but require_osd_release < quincy 2023-07-10T17:15:22.943 INFO:teuthology.orchestra.run.smithi116.stdout: all OSDs are running quincy or later but require_osd_release < quincy 2023-07-10T17:15:22.944 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 98/999 objects degraded (9.810%), 18 pgs degraded 2023-07-10T17:15:22.944 INFO:teuthology.orchestra.run.smithi116.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2023-07-10T17:15:22.944 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.4 is active+undersized+degraded, acting [1,0] 2023-07-10T17:15:22.944 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2023-07-10T17:15:22.944 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2023-07-10T17:15:22.944 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2023-07-10T17:15:22.944 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2023-07-10T17:15:22.945 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2023-07-10T17:15:22.945 INFO:teuthology.orchestra.run.smithi116.stdout: pg 2.1d is active+undersized+degraded, acting [6,0] 2023-07-10T17:15:22.945 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2023-07-10T17:15:22.945 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2023-07-10T17:15:22.945 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2023-07-10T17:15:22.945 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2023-07-10T17:15:22.945 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2023-07-10T17:15:22.945 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2023-07-10T17:15:22.946 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2023-07-10T17:15:22.946 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2023-07-10T17:15:22.946 INFO:teuthology.orchestra.run.smithi116.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2023-07-10T17:15:22.946 INFO:teuthology.orchestra.run.smithi116.stdout: pg 4.15 is active+undersized+degraded, acting [5,3] 2023-07-10T17:15:23.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:22 smithi192 bash[50362]: audit 2023-07-10T17:15:21.322221+0000 mgr.y (mgr.25390) 314 : audit 0 from='client.44572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:23.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:22 smithi192 bash[50362]: audit 2023-07-10T17:15:21.685502+0000 mgr.y (mgr.25390) 315 : audit 0 from='client.44578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:23.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:22 smithi192 bash[50362]: cluster 2023-07-10T17:15:21.718137+0000 mon.a (mon.0) 798 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2023-07-10T17:15:23.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:22 smithi192 bash[50362]: cluster 2023-07-10T17:15:22.123338+0000 mgr.y (mgr.25390) 316 : cluster 0 pgmap v230: 161 pgs: 32 peering, 19 active+undersized, 18 active+undersized+degraded, 92 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 98/999 objects degraded (9.810%) 2023-07-10T17:15:23.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:22 smithi192 bash[50362]: audit 2023-07-10T17:15:22.146115+0000 mon.a (mon.0) 799 : audit 0 from='client.34731 172.21.15.116:0/1850115851' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:23.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:22 smithi192 bash[50362]: cluster 2023-07-10T17:15:22.614561+0000 mon.a (mon.0) 800 : cluster 3 Health check update: Degraded data redundancy: 132/999 objects degraded (13.213%), 26 pgs degraded (PG_DEGRADED) 2023-07-10T17:15:23.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:22 smithi116 bash[65980]: audit 2023-07-10T17:15:21.322221+0000 mgr.y (mgr.25390) 314 : audit 0 from='client.44572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:23.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:22 smithi116 bash[65980]: audit 2023-07-10T17:15:21.685502+0000 mgr.y (mgr.25390) 315 : audit 0 from='client.44578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:23.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:22 smithi116 bash[65980]: cluster 2023-07-10T17:15:21.718137+0000 mon.a (mon.0) 798 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2023-07-10T17:15:23.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:22 smithi116 bash[65980]: cluster 2023-07-10T17:15:22.123338+0000 mgr.y (mgr.25390) 316 : cluster 0 pgmap v230: 161 pgs: 32 peering, 19 active+undersized, 18 active+undersized+degraded, 92 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 98/999 objects degraded (9.810%) 2023-07-10T17:15:23.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:22 smithi116 bash[65980]: audit 2023-07-10T17:15:22.146115+0000 mon.a (mon.0) 799 : audit 0 from='client.34731 172.21.15.116:0/1850115851' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:23.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:22 smithi116 bash[65980]: cluster 2023-07-10T17:15:22.614561+0000 mon.a (mon.0) 800 : cluster 3 Health check update: Degraded data redundancy: 132/999 objects degraded (13.213%), 26 pgs degraded (PG_DEGRADED) 2023-07-10T17:15:23.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:22 smithi116 bash[63900]: audit 2023-07-10T17:15:21.322221+0000 mgr.y (mgr.25390) 314 : audit 0 from='client.44572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:23.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:22 smithi116 bash[63900]: audit 2023-07-10T17:15:21.685502+0000 mgr.y (mgr.25390) 315 : audit 0 from='client.44578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:23.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:22 smithi116 bash[63900]: cluster 2023-07-10T17:15:21.718137+0000 mon.a (mon.0) 798 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2023-07-10T17:15:23.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:22 smithi116 bash[63900]: cluster 2023-07-10T17:15:22.123338+0000 mgr.y (mgr.25390) 316 : cluster 0 pgmap v230: 161 pgs: 32 peering, 19 active+undersized, 18 active+undersized+degraded, 92 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 98/999 objects degraded (9.810%) 2023-07-10T17:15:23.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:22 smithi116 bash[63900]: audit 2023-07-10T17:15:22.146115+0000 mon.a (mon.0) 799 : audit 0 from='client.34731 172.21.15.116:0/1850115851' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:23.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:22 smithi116 bash[63900]: cluster 2023-07-10T17:15:22.614561+0000 mon.a (mon.0) 800 : cluster 3 Health check update: Degraded data redundancy: 132/999 objects degraded (13.213%), 26 pgs degraded (PG_DEGRADED) 2023-07-10T17:15:24.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:23 smithi192 bash[50362]: audit 2023-07-10T17:15:22.508690+0000 mgr.y (mgr.25390) 317 : audit 0 from='client.34737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:24.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:23 smithi192 bash[50362]: audit 2023-07-10T17:15:22.938241+0000 mon.a (mon.0) 801 : audit 0 from='client.34743 172.21.15.116:0/2666600982' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:15:24.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:23 smithi116 bash[65980]: audit 2023-07-10T17:15:22.508690+0000 mgr.y (mgr.25390) 317 : audit 0 from='client.34737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:24.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:23 smithi116 bash[65980]: audit 2023-07-10T17:15:22.938241+0000 mon.a (mon.0) 801 : audit 0 from='client.34743 172.21.15.116:0/2666600982' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:15:24.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:23 smithi116 bash[63900]: audit 2023-07-10T17:15:22.508690+0000 mgr.y (mgr.25390) 317 : audit 0 from='client.34737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:24.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:23 smithi116 bash[63900]: audit 2023-07-10T17:15:22.938241+0000 mon.a (mon.0) 801 : audit 0 from='client.34743 172.21.15.116:0/2666600982' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:15:25.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:24 smithi192 bash[50362]: cluster 2023-07-10T17:15:24.124546+0000 mgr.y (mgr.25390) 318 : cluster 0 pgmap v231: 161 pgs: 32 peering, 15 active+undersized, 15 active+undersized+degraded, 99 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 85/999 objects degraded (8.509%) 2023-07-10T17:15:25.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:24 smithi116 bash[65980]: cluster 2023-07-10T17:15:24.124546+0000 mgr.y (mgr.25390) 318 : cluster 0 pgmap v231: 161 pgs: 32 peering, 15 active+undersized, 15 active+undersized+degraded, 99 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 85/999 objects degraded (8.509%) 2023-07-10T17:15:25.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:24 smithi116 bash[63900]: cluster 2023-07-10T17:15:24.124546+0000 mgr.y (mgr.25390) 318 : cluster 0 pgmap v231: 161 pgs: 32 peering, 15 active+undersized, 15 active+undersized+degraded, 99 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 85/999 objects degraded (8.509%) 2023-07-10T17:15:26.582 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:26 smithi192 bash[50362]: cluster 2023-07-10T17:15:26.177820+0000 mon.a (mon.0) 802 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 85/999 objects degraded (8.509%), 15 pgs degraded) 2023-07-10T17:15:26.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:26 smithi116 bash[65980]: cluster 2023-07-10T17:15:26.177820+0000 mon.a (mon.0) 802 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 85/999 objects degraded (8.509%), 15 pgs degraded) 2023-07-10T17:15:26.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:26 smithi116 bash[63900]: cluster 2023-07-10T17:15:26.177820+0000 mon.a (mon.0) 802 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 85/999 objects degraded (8.509%), 15 pgs degraded) 2023-07-10T17:15:27.514 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:27 smithi192 bash[50362]: cluster 2023-07-10T17:15:26.125933+0000 mgr.y (mgr.25390) 319 : cluster 0 pgmap v232: 161 pgs: 16 peering, 145 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 159 B/s rd, 0 op/s 2023-07-10T17:15:27.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:27 smithi116 bash[63900]: cluster 2023-07-10T17:15:26.125933+0000 mgr.y (mgr.25390) 319 : cluster 0 pgmap v232: 161 pgs: 16 peering, 145 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 159 B/s rd, 0 op/s 2023-07-10T17:15:27.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:27 smithi116 bash[65980]: cluster 2023-07-10T17:15:26.125933+0000 mgr.y (mgr.25390) 319 : cluster 0 pgmap v232: 161 pgs: 16 peering, 145 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 159 B/s rd, 0 op/s 2023-07-10T17:15:28.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:28 smithi192 bash[50362]: cluster 2023-07-10T17:15:28.127164+0000 mgr.y (mgr.25390) 320 : cluster 0 pgmap v233: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2023-07-10T17:15:28.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:28 smithi116 bash[65980]: cluster 2023-07-10T17:15:28.127164+0000 mgr.y (mgr.25390) 320 : cluster 0 pgmap v233: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2023-07-10T17:15:28.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:28 smithi116 bash[63900]: cluster 2023-07-10T17:15:28.127164+0000 mgr.y (mgr.25390) 320 : cluster 0 pgmap v233: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2023-07-10T17:15:30.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.555849+0000 mon.a (mon.0) 803 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:30.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.567779+0000 mon.a (mon.0) 804 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:30.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.569714+0000 mon.a (mon.0) 805 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:30.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.571379+0000 mon.a (mon.0) 806 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:15:30.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: cephadm 2023-07-10T17:15:29.572829+0000 mgr.y (mgr.25390) 321 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:15:30.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: cluster 2023-07-10T17:15:29.574075+0000 mgr.y (mgr.25390) 322 : cluster 0 pgmap v234: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 115 B/s rd, 0 op/s 2023-07-10T17:15:30.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: cluster 2023-07-10T17:15:29.574517+0000 mgr.y (mgr.25390) 323 : cluster 0 pgmap v235: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 130 B/s rd, 0 op/s 2023-07-10T17:15:30.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.584451+0000 mon.a (mon.0) 807 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:30.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.685772+0000 mon.a (mon.0) 808 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:15:30.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.690496+0000 mon.a (mon.0) 809 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:30.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.693197+0000 mon.a (mon.0) 810 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:30.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.695128+0000 mon.a (mon.0) 811 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:30.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.698438+0000 mon.a (mon.0) 812 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:30.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: cephadm 2023-07-10T17:15:29.699660+0000 mgr.y (mgr.25390) 324 : cephadm 1 Upgrade: Setting container_image for all osd 2023-07-10T17:15:30.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.709705+0000 mon.a (mon.0) 813 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:30.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.711494+0000 mon.a (mon.0) 814 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2023-07-10T17:15:30.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.722169+0000 mon.a (mon.0) 815 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2023-07-10T17:15:30.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.723640+0000 mon.a (mon.0) 816 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2023-07-10T17:15:30.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.729863+0000 mon.a (mon.0) 817 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2023-07-10T17:15:30.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.731226+0000 mon.a (mon.0) 818 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2023-07-10T17:15:30.898 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.737214+0000 mon.a (mon.0) 819 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2023-07-10T17:15:30.898 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.738536+0000 mon.a (mon.0) 820 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2023-07-10T17:15:30.898 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.746225+0000 mon.a (mon.0) 821 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2023-07-10T17:15:30.898 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.747479+0000 mon.a (mon.0) 822 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2023-07-10T17:15:30.898 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.752813+0000 mon.a (mon.0) 823 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2023-07-10T17:15:30.898 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.753969+0000 mon.a (mon.0) 824 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2023-07-10T17:15:30.898 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.762342+0000 mon.a (mon.0) 825 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2023-07-10T17:15:30.899 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.763542+0000 mon.a (mon.0) 826 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2023-07-10T17:15:30.899 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.768806+0000 mon.a (mon.0) 827 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2023-07-10T17:15:30.899 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.769936+0000 mon.a (mon.0) 828 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2023-07-10T17:15:30.899 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.774761+0000 mon.a (mon.0) 829 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2023-07-10T17:15:30.899 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: cephadm 2023-07-10T17:15:29.777518+0000 mgr.y (mgr.25390) 325 : cephadm 1 Upgrade: Setting require_osd_release to 18 reef 2023-07-10T17:15:30.899 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:30 smithi192 bash[50362]: audit 2023-07-10T17:15:29.777893+0000 mon.a (mon.0) 830 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2023-07-10T17:15:30.900 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.555849+0000 mon.a (mon.0) 803 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:30.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.567779+0000 mon.a (mon.0) 804 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:30.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.569714+0000 mon.a (mon.0) 805 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:30.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.571379+0000 mon.a (mon.0) 806 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:15:30.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: cephadm 2023-07-10T17:15:29.572829+0000 mgr.y (mgr.25390) 321 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:15:30.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: cluster 2023-07-10T17:15:29.574075+0000 mgr.y (mgr.25390) 322 : cluster 0 pgmap v234: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 115 B/s rd, 0 op/s 2023-07-10T17:15:30.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: cluster 2023-07-10T17:15:29.574517+0000 mgr.y (mgr.25390) 323 : cluster 0 pgmap v235: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 130 B/s rd, 0 op/s 2023-07-10T17:15:30.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.584451+0000 mon.a (mon.0) 807 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:30.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.685772+0000 mon.a (mon.0) 808 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:15:30.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.690496+0000 mon.a (mon.0) 809 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:30.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.693197+0000 mon.a (mon.0) 810 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:30.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.695128+0000 mon.a (mon.0) 811 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:30.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.698438+0000 mon.a (mon.0) 812 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:30.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: cephadm 2023-07-10T17:15:29.699660+0000 mgr.y (mgr.25390) 324 : cephadm 1 Upgrade: Setting container_image for all osd 2023-07-10T17:15:30.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.709705+0000 mon.a (mon.0) 813 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:30.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.711494+0000 mon.a (mon.0) 814 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2023-07-10T17:15:30.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.722169+0000 mon.a (mon.0) 815 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2023-07-10T17:15:30.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.723640+0000 mon.a (mon.0) 816 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2023-07-10T17:15:30.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.729863+0000 mon.a (mon.0) 817 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2023-07-10T17:15:30.906 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.555849+0000 mon.a (mon.0) 803 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:30.906 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.567779+0000 mon.a (mon.0) 804 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:30.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.569714+0000 mon.a (mon.0) 805 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:30.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.571379+0000 mon.a (mon.0) 806 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:15:30.909 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: cephadm 2023-07-10T17:15:29.572829+0000 mgr.y (mgr.25390) 321 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:15:30.909 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: cluster 2023-07-10T17:15:29.574075+0000 mgr.y (mgr.25390) 322 : cluster 0 pgmap v234: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 115 B/s rd, 0 op/s 2023-07-10T17:15:30.909 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: cluster 2023-07-10T17:15:29.574517+0000 mgr.y (mgr.25390) 323 : cluster 0 pgmap v235: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 130 B/s rd, 0 op/s 2023-07-10T17:15:30.909 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.584451+0000 mon.a (mon.0) 807 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:30.909 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.685772+0000 mon.a (mon.0) 808 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:15:30.910 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.690496+0000 mon.a (mon.0) 809 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:30.910 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.693197+0000 mon.a (mon.0) 810 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:30.910 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.695128+0000 mon.a (mon.0) 811 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:30.910 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.698438+0000 mon.a (mon.0) 812 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:30.910 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: cephadm 2023-07-10T17:15:29.699660+0000 mgr.y (mgr.25390) 324 : cephadm 1 Upgrade: Setting container_image for all osd 2023-07-10T17:15:30.910 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.709705+0000 mon.a (mon.0) 813 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:30.911 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.711494+0000 mon.a (mon.0) 814 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2023-07-10T17:15:30.911 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.722169+0000 mon.a (mon.0) 815 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2023-07-10T17:15:30.911 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.723640+0000 mon.a (mon.0) 816 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2023-07-10T17:15:30.911 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.729863+0000 mon.a (mon.0) 817 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2023-07-10T17:15:30.911 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.731226+0000 mon.a (mon.0) 818 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2023-07-10T17:15:30.913 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.737214+0000 mon.a (mon.0) 819 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2023-07-10T17:15:30.913 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.738536+0000 mon.a (mon.0) 820 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2023-07-10T17:15:30.913 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.746225+0000 mon.a (mon.0) 821 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2023-07-10T17:15:30.913 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.747479+0000 mon.a (mon.0) 822 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2023-07-10T17:15:30.913 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.752813+0000 mon.a (mon.0) 823 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2023-07-10T17:15:30.913 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.753969+0000 mon.a (mon.0) 824 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2023-07-10T17:15:30.913 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.762342+0000 mon.a (mon.0) 825 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2023-07-10T17:15:30.914 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.763542+0000 mon.a (mon.0) 826 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2023-07-10T17:15:30.914 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.768806+0000 mon.a (mon.0) 827 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2023-07-10T17:15:30.914 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.769936+0000 mon.a (mon.0) 828 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2023-07-10T17:15:30.914 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.774761+0000 mon.a (mon.0) 829 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2023-07-10T17:15:30.914 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: cephadm 2023-07-10T17:15:29.777518+0000 mgr.y (mgr.25390) 325 : cephadm 1 Upgrade: Setting require_osd_release to 18 reef 2023-07-10T17:15:30.914 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[63900]: audit 2023-07-10T17:15:29.777893+0000 mon.a (mon.0) 830 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2023-07-10T17:15:30.915 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.731226+0000 mon.a (mon.0) 818 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2023-07-10T17:15:30.915 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.737214+0000 mon.a (mon.0) 819 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2023-07-10T17:15:30.915 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.738536+0000 mon.a (mon.0) 820 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2023-07-10T17:15:30.915 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.746225+0000 mon.a (mon.0) 821 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2023-07-10T17:15:30.915 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.747479+0000 mon.a (mon.0) 822 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2023-07-10T17:15:30.915 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.752813+0000 mon.a (mon.0) 823 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2023-07-10T17:15:30.916 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.753969+0000 mon.a (mon.0) 824 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2023-07-10T17:15:30.916 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.762342+0000 mon.a (mon.0) 825 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2023-07-10T17:15:30.916 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.763542+0000 mon.a (mon.0) 826 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2023-07-10T17:15:30.916 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.768806+0000 mon.a (mon.0) 827 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2023-07-10T17:15:30.916 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.769936+0000 mon.a (mon.0) 828 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2023-07-10T17:15:30.916 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.774761+0000 mon.a (mon.0) 829 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2023-07-10T17:15:30.916 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: cephadm 2023-07-10T17:15:29.777518+0000 mgr.y (mgr.25390) 325 : cephadm 1 Upgrade: Setting require_osd_release to 18 reef 2023-07-10T17:15:30.917 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:30 smithi116 bash[65980]: audit 2023-07-10T17:15:29.777893+0000 mon.a (mon.0) 830 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2023-07-10T17:15:31.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:31 smithi192 bash[50362]: cluster 2023-07-10T17:15:30.772034+0000 mon.a (mon.0) 831 : cluster 1 Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2023-07-10T17:15:31.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:31 smithi192 bash[50362]: audit 2023-07-10T17:15:30.781602+0000 mon.a (mon.0) 832 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2023-07-10T17:15:31.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:31 smithi192 bash[50362]: cluster 2023-07-10T17:15:30.781790+0000 mon.a (mon.0) 833 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2023-07-10T17:15:31.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:31 smithi192 bash[50362]: audit 2023-07-10T17:15:30.784970+0000 mon.a (mon.0) 834 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:31.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:31 smithi192 bash[50362]: cephadm 2023-07-10T17:15:30.786222+0000 mgr.y (mgr.25390) 326 : cephadm 1 Upgrade: Setting container_image for all mds 2023-07-10T17:15:31.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:31 smithi192 bash[50362]: audit 2023-07-10T17:15:30.793181+0000 mon.a (mon.0) 835 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:31.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:31 smithi116 bash[63900]: cluster 2023-07-10T17:15:30.772034+0000 mon.a (mon.0) 831 : cluster 1 Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2023-07-10T17:15:31.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:31 smithi116 bash[63900]: audit 2023-07-10T17:15:30.781602+0000 mon.a (mon.0) 832 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2023-07-10T17:15:31.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:31 smithi116 bash[63900]: cluster 2023-07-10T17:15:30.781790+0000 mon.a (mon.0) 833 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2023-07-10T17:15:31.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:31 smithi116 bash[63900]: audit 2023-07-10T17:15:30.784970+0000 mon.a (mon.0) 834 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:31.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:31 smithi116 bash[63900]: cephadm 2023-07-10T17:15:30.786222+0000 mgr.y (mgr.25390) 326 : cephadm 1 Upgrade: Setting container_image for all mds 2023-07-10T17:15:31.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:31 smithi116 bash[63900]: audit 2023-07-10T17:15:30.793181+0000 mon.a (mon.0) 835 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:31.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:31 smithi116 bash[65980]: cluster 2023-07-10T17:15:30.772034+0000 mon.a (mon.0) 831 : cluster 1 Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2023-07-10T17:15:31.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:31 smithi116 bash[65980]: audit 2023-07-10T17:15:30.781602+0000 mon.a (mon.0) 832 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2023-07-10T17:15:31.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:31 smithi116 bash[65980]: cluster 2023-07-10T17:15:30.781790+0000 mon.a (mon.0) 833 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2023-07-10T17:15:31.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:31 smithi116 bash[65980]: audit 2023-07-10T17:15:30.784970+0000 mon.a (mon.0) 834 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:31.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:31 smithi116 bash[65980]: cephadm 2023-07-10T17:15:30.786222+0000 mgr.y (mgr.25390) 326 : cephadm 1 Upgrade: Setting container_image for all mds 2023-07-10T17:15:31.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:31 smithi116 bash[65980]: audit 2023-07-10T17:15:30.793181+0000 mon.a (mon.0) 835 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:32.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:32 smithi116 bash[63900]: cluster 2023-07-10T17:15:31.575507+0000 mgr.y (mgr.25390) 327 : cluster 0 pgmap v237: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 137 B/s rd, 0 op/s 2023-07-10T17:15:32.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:32 smithi116 bash[63900]: audit 2023-07-10T17:15:31.690340+0000 mon.a (mon.0) 836 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:32.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:32 smithi116 bash[65980]: cluster 2023-07-10T17:15:31.575507+0000 mgr.y (mgr.25390) 327 : cluster 0 pgmap v237: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 137 B/s rd, 0 op/s 2023-07-10T17:15:32.897 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:32 smithi116 bash[65980]: audit 2023-07-10T17:15:31.690340+0000 mon.a (mon.0) 836 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:33.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:32 smithi192 bash[50362]: cluster 2023-07-10T17:15:31.575507+0000 mgr.y (mgr.25390) 327 : cluster 0 pgmap v237: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 137 B/s rd, 0 op/s 2023-07-10T17:15:33.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:32 smithi192 bash[50362]: audit 2023-07-10T17:15:31.690340+0000 mon.a (mon.0) 836 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:34.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:34 smithi192 bash[50362]: cephadm 2023-07-10T17:15:33.366317+0000 mgr.y (mgr.25390) 328 : cephadm 1 Upgrade: Updating rgw.foo.smithi116.stnxgn (1/4) 2023-07-10T17:15:34.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:34 smithi192 bash[50362]: audit 2023-07-10T17:15:33.376867+0000 mon.a (mon.0) 837 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:34.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:34 smithi192 bash[50362]: audit 2023-07-10T17:15:33.379092+0000 mon.a (mon.0) 838 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi116.stnxgn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:15:34.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:34 smithi192 bash[50362]: audit 2023-07-10T17:15:33.382726+0000 mon.a (mon.0) 839 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:34.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:34 smithi192 bash[50362]: cephadm 2023-07-10T17:15:33.384220+0000 mgr.y (mgr.25390) 329 : cephadm 1 Deploying daemon rgw.foo.smithi116.stnxgn on smithi116 2023-07-10T17:15:34.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:34 smithi192 bash[50362]: cluster 2023-07-10T17:15:33.576675+0000 mgr.y (mgr.25390) 330 : cluster 0 pgmap v238: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:34.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:34 smithi116 bash[63900]: cephadm 2023-07-10T17:15:33.366317+0000 mgr.y (mgr.25390) 328 : cephadm 1 Upgrade: Updating rgw.foo.smithi116.stnxgn (1/4) 2023-07-10T17:15:34.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:34 smithi116 bash[63900]: audit 2023-07-10T17:15:33.376867+0000 mon.a (mon.0) 837 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:34.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:34 smithi116 bash[63900]: audit 2023-07-10T17:15:33.379092+0000 mon.a (mon.0) 838 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi116.stnxgn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:15:34.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:34 smithi116 bash[63900]: audit 2023-07-10T17:15:33.382726+0000 mon.a (mon.0) 839 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:34.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:34 smithi116 bash[63900]: cephadm 2023-07-10T17:15:33.384220+0000 mgr.y (mgr.25390) 329 : cephadm 1 Deploying daemon rgw.foo.smithi116.stnxgn on smithi116 2023-07-10T17:15:34.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:34 smithi116 bash[63900]: cluster 2023-07-10T17:15:33.576675+0000 mgr.y (mgr.25390) 330 : cluster 0 pgmap v238: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:34.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:34 smithi116 bash[65980]: cephadm 2023-07-10T17:15:33.366317+0000 mgr.y (mgr.25390) 328 : cephadm 1 Upgrade: Updating rgw.foo.smithi116.stnxgn (1/4) 2023-07-10T17:15:34.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:34 smithi116 bash[65980]: audit 2023-07-10T17:15:33.376867+0000 mon.a (mon.0) 837 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:34.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:34 smithi116 bash[65980]: audit 2023-07-10T17:15:33.379092+0000 mon.a (mon.0) 838 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi116.stnxgn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:15:34.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:34 smithi116 bash[65980]: audit 2023-07-10T17:15:33.382726+0000 mon.a (mon.0) 839 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:34.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:34 smithi116 bash[65980]: cephadm 2023-07-10T17:15:33.384220+0000 mgr.y (mgr.25390) 329 : cephadm 1 Deploying daemon rgw.foo.smithi116.stnxgn on smithi116 2023-07-10T17:15:34.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:34 smithi116 bash[65980]: cluster 2023-07-10T17:15:33.576675+0000 mgr.y (mgr.25390) 330 : cluster 0 pgmap v238: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:36.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:36 smithi116 bash[63900]: cluster 2023-07-10T17:15:35.577580+0000 mgr.y (mgr.25390) 331 : cluster 0 pgmap v239: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:36.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:36 smithi116 bash[65980]: cluster 2023-07-10T17:15:35.577580+0000 mgr.y (mgr.25390) 331 : cluster 0 pgmap v239: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:37.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:36 smithi192 bash[50362]: cluster 2023-07-10T17:15:35.577580+0000 mgr.y (mgr.25390) 331 : cluster 0 pgmap v239: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:38.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:38 smithi192 bash[50362]: audit 2023-07-10T17:15:37.224329+0000 mon.a (mon.0) 840 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:38.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:38 smithi192 bash[50362]: audit 2023-07-10T17:15:37.233736+0000 mon.a (mon.0) 841 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:38.661 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:38 smithi116 bash[63900]: audit 2023-07-10T17:15:37.224329+0000 mon.a (mon.0) 840 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:38.662 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:38 smithi116 bash[63900]: audit 2023-07-10T17:15:37.233736+0000 mon.a (mon.0) 841 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:38.662 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:38 smithi116 bash[65980]: audit 2023-07-10T17:15:37.224329+0000 mon.a (mon.0) 840 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:38.662 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:38 smithi116 bash[65980]: audit 2023-07-10T17:15:37.233736+0000 mon.a (mon.0) 841 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:39.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:39 smithi192 bash[50362]: cluster 2023-07-10T17:15:37.579157+0000 mgr.y (mgr.25390) 332 : cluster 0 pgmap v240: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:39.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:39 smithi116 bash[63900]: cluster 2023-07-10T17:15:37.579157+0000 mgr.y (mgr.25390) 332 : cluster 0 pgmap v240: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:39.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:39 smithi116 bash[65980]: cluster 2023-07-10T17:15:37.579157+0000 mgr.y (mgr.25390) 332 : cluster 0 pgmap v240: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:15:41.312 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:40 smithi116 bash[63900]: cluster 2023-07-10T17:15:39.580060+0000 mgr.y (mgr.25390) 333 : cluster 0 pgmap v241: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 1 op/s 2023-07-10T17:15:41.313 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:40 smithi116 bash[63900]: cephadm 2023-07-10T17:15:39.977160+0000 mgr.y (mgr.25390) 334 : cephadm 1 Upgrade: Updating rgw.smpl.smithi116.vpoihf (2/4) 2023-07-10T17:15:41.313 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:40 smithi116 bash[63900]: audit 2023-07-10T17:15:39.988039+0000 mon.a (mon.0) 842 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:41.313 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:40 smithi116 bash[63900]: audit 2023-07-10T17:15:39.989777+0000 mon.a (mon.0) 843 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi116.vpoihf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:15:41.313 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:40 smithi116 bash[63900]: audit 2023-07-10T17:15:39.992613+0000 mon.a (mon.0) 844 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:41.313 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:40 smithi116 bash[63900]: cephadm 2023-07-10T17:15:39.994215+0000 mgr.y (mgr.25390) 335 : cephadm 1 Deploying daemon rgw.smpl.smithi116.vpoihf on smithi116 2023-07-10T17:15:41.313 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:40 smithi116 bash[65980]: cluster 2023-07-10T17:15:39.580060+0000 mgr.y (mgr.25390) 333 : cluster 0 pgmap v241: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 1 op/s 2023-07-10T17:15:41.314 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:40 smithi116 bash[65980]: cephadm 2023-07-10T17:15:39.977160+0000 mgr.y (mgr.25390) 334 : cephadm 1 Upgrade: Updating rgw.smpl.smithi116.vpoihf (2/4) 2023-07-10T17:15:41.314 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:40 smithi116 bash[65980]: audit 2023-07-10T17:15:39.988039+0000 mon.a (mon.0) 842 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:41.314 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:40 smithi116 bash[65980]: audit 2023-07-10T17:15:39.989777+0000 mon.a (mon.0) 843 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi116.vpoihf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:15:41.314 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:40 smithi116 bash[65980]: audit 2023-07-10T17:15:39.992613+0000 mon.a (mon.0) 844 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:41.314 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:40 smithi116 bash[65980]: cephadm 2023-07-10T17:15:39.994215+0000 mgr.y (mgr.25390) 335 : cephadm 1 Deploying daemon rgw.smpl.smithi116.vpoihf on smithi116 2023-07-10T17:15:41.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:40 smithi192 bash[50362]: cluster 2023-07-10T17:15:39.580060+0000 mgr.y (mgr.25390) 333 : cluster 0 pgmap v241: 161 pgs: 161 active+clean; 604 KiB data, 296 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 1 op/s 2023-07-10T17:15:41.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:40 smithi192 bash[50362]: cephadm 2023-07-10T17:15:39.977160+0000 mgr.y (mgr.25390) 334 : cephadm 1 Upgrade: Updating rgw.smpl.smithi116.vpoihf (2/4) 2023-07-10T17:15:41.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:40 smithi192 bash[50362]: audit 2023-07-10T17:15:39.988039+0000 mon.a (mon.0) 842 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:41.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:40 smithi192 bash[50362]: audit 2023-07-10T17:15:39.989777+0000 mon.a (mon.0) 843 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi116.vpoihf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:15:41.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:40 smithi192 bash[50362]: audit 2023-07-10T17:15:39.992613+0000 mon.a (mon.0) 844 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:41.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:40 smithi192 bash[50362]: cephadm 2023-07-10T17:15:39.994215+0000 mgr.y (mgr.25390) 335 : cephadm 1 Deploying daemon rgw.smpl.smithi116.vpoihf on smithi116 2023-07-10T17:15:42.248 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:41 smithi116 bash[63900]: audit 2023-07-10T17:15:41.841045+0000 mon.a (mon.0) 845 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1e"}]: dispatch 2023-07-10T17:15:42.248 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:41 smithi116 bash[65980]: audit 2023-07-10T17:15:41.841045+0000 mon.a (mon.0) 845 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1e"}]: dispatch 2023-07-10T17:15:42.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:41 smithi192 bash[50362]: audit 2023-07-10T17:15:41.841045+0000 mon.a (mon.0) 845 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1e"}]: dispatch 2023-07-10T17:15:43.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:42 smithi116 bash[63900]: cluster 2023-07-10T17:15:41.581416+0000 mgr.y (mgr.25390) 336 : cluster 0 pgmap v242: 161 pgs: 161 active+clean; 604 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 189 B/s wr, 54 op/s 2023-07-10T17:15:43.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:42 smithi116 bash[63900]: audit 2023-07-10T17:15:41.994717+0000 mon.a (mon.0) 846 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1e"}]': finished 2023-07-10T17:15:43.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:42 smithi116 bash[63900]: cluster 2023-07-10T17:15:41.994936+0000 mon.a (mon.0) 847 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2023-07-10T17:15:43.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:42 smithi116 bash[65980]: cluster 2023-07-10T17:15:41.581416+0000 mgr.y (mgr.25390) 336 : cluster 0 pgmap v242: 161 pgs: 161 active+clean; 604 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 189 B/s wr, 54 op/s 2023-07-10T17:15:43.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:42 smithi116 bash[65980]: audit 2023-07-10T17:15:41.994717+0000 mon.a (mon.0) 846 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1e"}]': finished 2023-07-10T17:15:43.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:42 smithi116 bash[65980]: cluster 2023-07-10T17:15:41.994936+0000 mon.a (mon.0) 847 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2023-07-10T17:15:43.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:42 smithi192 bash[50362]: cluster 2023-07-10T17:15:41.581416+0000 mgr.y (mgr.25390) 336 : cluster 0 pgmap v242: 161 pgs: 161 active+clean; 604 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 189 B/s wr, 54 op/s 2023-07-10T17:15:43.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:42 smithi192 bash[50362]: audit 2023-07-10T17:15:41.994717+0000 mon.a (mon.0) 846 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1e"}]': finished 2023-07-10T17:15:43.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:42 smithi192 bash[50362]: cluster 2023-07-10T17:15:41.994936+0000 mon.a (mon.0) 847 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2023-07-10T17:15:44.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:44 smithi192 bash[50362]: cluster 2023-07-10T17:15:43.004762+0000 mon.a (mon.0) 848 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2023-07-10T17:15:44.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:44 smithi116 bash[63900]: cluster 2023-07-10T17:15:43.004762+0000 mon.a (mon.0) 848 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2023-07-10T17:15:44.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:44 smithi116 bash[65980]: cluster 2023-07-10T17:15:43.004762+0000 mon.a (mon.0) 848 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2023-07-10T17:15:45.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:45 smithi192 bash[50362]: cluster 2023-07-10T17:15:43.582245+0000 mgr.y (mgr.25390) 337 : cluster 0 pgmap v245: 161 pgs: 161 active+clean; 604 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 255 B/s wr, 130 op/s 2023-07-10T17:15:45.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:45 smithi192 bash[50362]: audit 2023-07-10T17:15:44.004663+0000 mon.a (mon.0) 849 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:45.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:45 smithi192 bash[50362]: audit 2023-07-10T17:15:44.017451+0000 mon.a (mon.0) 850 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:45.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:45 smithi116 bash[63900]: cluster 2023-07-10T17:15:43.582245+0000 mgr.y (mgr.25390) 337 : cluster 0 pgmap v245: 161 pgs: 161 active+clean; 604 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 255 B/s wr, 130 op/s 2023-07-10T17:15:45.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:45 smithi116 bash[63900]: audit 2023-07-10T17:15:44.004663+0000 mon.a (mon.0) 849 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:45.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:45 smithi116 bash[63900]: audit 2023-07-10T17:15:44.017451+0000 mon.a (mon.0) 850 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:45.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:45 smithi116 bash[65980]: cluster 2023-07-10T17:15:43.582245+0000 mgr.y (mgr.25390) 337 : cluster 0 pgmap v245: 161 pgs: 161 active+clean; 604 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 255 B/s wr, 130 op/s 2023-07-10T17:15:45.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:45 smithi116 bash[65980]: audit 2023-07-10T17:15:44.004663+0000 mon.a (mon.0) 849 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:45.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:45 smithi116 bash[65980]: audit 2023-07-10T17:15:44.017451+0000 mon.a (mon.0) 850 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:47.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:47 smithi192 bash[50362]: cluster 2023-07-10T17:15:45.582999+0000 mgr.y (mgr.25390) 338 : cluster 0 pgmap v246: 161 pgs: 161 active+clean; 604 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 255 B/s wr, 155 op/s 2023-07-10T17:15:47.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:47 smithi192 bash[50362]: audit 2023-07-10T17:15:46.648771+0000 mon.a (mon.0) 851 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:47.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:47 smithi192 bash[50362]: audit 2023-07-10T17:15:46.650644+0000 mon.a (mon.0) 852 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.myrhsb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:15:47.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:47 smithi192 bash[50362]: audit 2023-07-10T17:15:46.654100+0000 mon.a (mon.0) 853 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:47.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:47 smithi116 bash[63900]: cluster 2023-07-10T17:15:45.582999+0000 mgr.y (mgr.25390) 338 : cluster 0 pgmap v246: 161 pgs: 161 active+clean; 604 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 255 B/s wr, 155 op/s 2023-07-10T17:15:47.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:47 smithi116 bash[63900]: audit 2023-07-10T17:15:46.648771+0000 mon.a (mon.0) 851 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:47.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:47 smithi116 bash[63900]: audit 2023-07-10T17:15:46.650644+0000 mon.a (mon.0) 852 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.myrhsb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:15:47.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:47 smithi116 bash[63900]: audit 2023-07-10T17:15:46.654100+0000 mon.a (mon.0) 853 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:47.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:47 smithi116 bash[65980]: cluster 2023-07-10T17:15:45.582999+0000 mgr.y (mgr.25390) 338 : cluster 0 pgmap v246: 161 pgs: 161 active+clean; 604 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 255 B/s wr, 155 op/s 2023-07-10T17:15:47.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:47 smithi116 bash[65980]: audit 2023-07-10T17:15:46.648771+0000 mon.a (mon.0) 851 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:47.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:47 smithi116 bash[65980]: audit 2023-07-10T17:15:46.650644+0000 mon.a (mon.0) 852 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.myrhsb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:15:47.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:47 smithi116 bash[65980]: audit 2023-07-10T17:15:46.654100+0000 mon.a (mon.0) 853 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:48.211 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:48 smithi192 bash[50362]: cephadm 2023-07-10T17:15:46.639456+0000 mgr.y (mgr.25390) 339 : cephadm 1 Upgrade: Updating rgw.foo.smithi192.myrhsb (3/4) 2023-07-10T17:15:48.211 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:48 smithi192 bash[50362]: cephadm 2023-07-10T17:15:46.655792+0000 mgr.y (mgr.25390) 340 : cephadm 1 Deploying daemon rgw.foo.smithi192.myrhsb on smithi192 2023-07-10T17:15:48.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:48 smithi116 bash[65980]: cephadm 2023-07-10T17:15:46.639456+0000 mgr.y (mgr.25390) 339 : cephadm 1 Upgrade: Updating rgw.foo.smithi192.myrhsb (3/4) 2023-07-10T17:15:48.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:48 smithi116 bash[65980]: cephadm 2023-07-10T17:15:46.655792+0000 mgr.y (mgr.25390) 340 : cephadm 1 Deploying daemon rgw.foo.smithi192.myrhsb on smithi192 2023-07-10T17:15:48.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:48 smithi116 bash[63900]: cephadm 2023-07-10T17:15:46.639456+0000 mgr.y (mgr.25390) 339 : cephadm 1 Upgrade: Updating rgw.foo.smithi192.myrhsb (3/4) 2023-07-10T17:15:48.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:48 smithi116 bash[63900]: cephadm 2023-07-10T17:15:46.655792+0000 mgr.y (mgr.25390) 340 : cephadm 1 Deploying daemon rgw.foo.smithi192.myrhsb on smithi192 2023-07-10T17:15:49.319 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:49 smithi192 bash[50362]: cluster 2023-07-10T17:15:47.584474+0000 mgr.y (mgr.25390) 341 : cluster 0 pgmap v247: 161 pgs: 161 active+clean; 604 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 383 B/s wr, 226 op/s 2023-07-10T17:15:49.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:49 smithi116 bash[65980]: cluster 2023-07-10T17:15:47.584474+0000 mgr.y (mgr.25390) 341 : cluster 0 pgmap v247: 161 pgs: 161 active+clean; 604 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 383 B/s wr, 226 op/s 2023-07-10T17:15:49.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:49 smithi116 bash[63900]: cluster 2023-07-10T17:15:47.584474+0000 mgr.y (mgr.25390) 341 : cluster 0 pgmap v247: 161 pgs: 161 active+clean; 604 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 383 B/s wr, 226 op/s 2023-07-10T17:15:51.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:51 smithi192 bash[50362]: cluster 2023-07-10T17:15:49.585462+0000 mgr.y (mgr.25390) 342 : cluster 0 pgmap v248: 161 pgs: 161 active+clean; 604 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 127 B/s wr, 171 op/s 2023-07-10T17:15:51.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:51 smithi192 bash[50362]: audit 2023-07-10T17:15:50.400294+0000 mon.a (mon.0) 854 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:51.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:51 smithi192 bash[50362]: audit 2023-07-10T17:15:50.411201+0000 mon.a (mon.0) 855 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:51.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:51 smithi116 bash[63900]: cluster 2023-07-10T17:15:49.585462+0000 mgr.y (mgr.25390) 342 : cluster 0 pgmap v248: 161 pgs: 161 active+clean; 604 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 127 B/s wr, 171 op/s 2023-07-10T17:15:51.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:51 smithi116 bash[63900]: audit 2023-07-10T17:15:50.400294+0000 mon.a (mon.0) 854 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:51.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:51 smithi116 bash[63900]: audit 2023-07-10T17:15:50.411201+0000 mon.a (mon.0) 855 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:51.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:51 smithi116 bash[65980]: cluster 2023-07-10T17:15:49.585462+0000 mgr.y (mgr.25390) 342 : cluster 0 pgmap v248: 161 pgs: 161 active+clean; 604 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 127 B/s wr, 171 op/s 2023-07-10T17:15:51.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:51 smithi116 bash[65980]: audit 2023-07-10T17:15:50.400294+0000 mon.a (mon.0) 854 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:51.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:51 smithi116 bash[65980]: audit 2023-07-10T17:15:50.411201+0000 mon.a (mon.0) 855 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:53.350 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:15:53.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:53 smithi192 bash[50362]: cluster 2023-07-10T17:15:51.586793+0000 mgr.y (mgr.25390) 343 : cluster 0 pgmap v249: 161 pgs: 161 active+clean; 604 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 106 B/s wr, 144 op/s 2023-07-10T17:15:53.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:53 smithi116 bash[63900]: cluster 2023-07-10T17:15:51.586793+0000 mgr.y (mgr.25390) 343 : cluster 0 pgmap v249: 161 pgs: 161 active+clean; 604 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 106 B/s wr, 144 op/s 2023-07-10T17:15:53.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:53 smithi116 bash[65980]: cluster 2023-07-10T17:15:51.586793+0000 mgr.y (mgr.25390) 343 : cluster 0 pgmap v249: 161 pgs: 161 active+clean; 604 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 106 B/s wr, 144 op/s 2023-07-10T17:15:54.099 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:15:54.099 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (9m) 2m ago 15m 10.9M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:15:54.100 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (7m) 35s ago 15m 26.0M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:15:54.100 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 error 2m ago 12m - - 2023-07-10T17:15:54.100 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (6m) 35s ago 19m 430M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:15:54.100 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (7m) 2m ago 21m 475M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:15:54.100 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (6m) 2m ago 21m 59.6M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:15:54.100 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (5m) 35s ago 19m 55.4M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 74a082a2e795 2023-07-10T17:15:54.100 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (5m) 2m ago 20m 49.9M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:15:54.101 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (9m) 2m ago 15m 9371k - 0.18.1 e5a616e4b9cf 3bd058d91353 2023-07-10T17:15:54.101 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (8m) 35s ago 15m 3944k - 0.18.1 e5a616e4b9cf 63551615c679 2023-07-10T17:15:54.101 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (4m) 2m ago 19m 53.2M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 0562f587fbca 2023-07-10T17:15:54.101 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (3m) 2m ago 18m 51.6M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 5b2a12a0bfc9 2023-07-10T17:15:54.101 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (3m) 2m ago 18m 47.9M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 88096bc0faa9 2023-07-10T17:15:54.101 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (2m) 2m ago 17m - 4096M 18.1.2-30-g8507faa7 17e8f0d79247 61ae280c471b 2023-07-10T17:15:54.101 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (2m) 35s ago 17m 54.3M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 44d809c18118 2023-07-10T17:15:54.102 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (96s) 35s ago 17m 50.2M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 179270eb7044 2023-07-10T17:15:54.102 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (67s) 35s ago 16m 50.5M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 b680bfbe9a0c 2023-07-10T17:15:54.102 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (39s) 35s ago 16m - 4096M 18.1.2-30-g8507faa7 17e8f0d79247 bd3788803e68 2023-07-10T17:15:54.102 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 error 35s ago 15m - - 2023-07-10T17:15:54.102 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 starting - - - - 2023-07-10T17:15:54.102 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 starting - - - - 2023-07-10T17:15:54.102 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 starting - - - - 2023-07-10T17:15:54.102 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (13m) 35s ago 13m 58.0M - 16.2.0 24ecd6d5f14c 65047e31745f 2023-07-10T17:15:54.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:54 smithi192 bash[50362]: cephadm 2023-07-10T17:15:53.199145+0000 mgr.y (mgr.25390) 344 : cephadm 1 Upgrade: Updating rgw.smpl.smithi192.yqkmva (4/4) 2023-07-10T17:15:54.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:54 smithi192 bash[50362]: audit 2023-07-10T17:15:53.208013+0000 mon.a (mon.0) 856 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:54.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:54 smithi192 bash[50362]: audit 2023-07-10T17:15:53.209546+0000 mon.a (mon.0) 857 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.yqkmva", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:15:54.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:54 smithi192 bash[50362]: audit 2023-07-10T17:15:53.211941+0000 mon.a (mon.0) 858 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:54.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:54 smithi192 bash[50362]: cephadm 2023-07-10T17:15:53.213022+0000 mgr.y (mgr.25390) 345 : cephadm 1 Deploying daemon rgw.smpl.smithi192.yqkmva on smithi192 2023-07-10T17:15:54.530 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:15:54.530 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:15:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:15:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:15:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:15:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:15:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:15:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:15:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 8 2023-07-10T17:15:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:15:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:15:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2023-07-10T17:15:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:15:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:15:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:15:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2023-07-10T17:15:54.533 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 16 2023-07-10T17:15:54.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:15:54.533 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:15:54.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:54 smithi116 bash[65980]: cephadm 2023-07-10T17:15:53.199145+0000 mgr.y (mgr.25390) 344 : cephadm 1 Upgrade: Updating rgw.smpl.smithi192.yqkmva (4/4) 2023-07-10T17:15:54.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:54 smithi116 bash[65980]: audit 2023-07-10T17:15:53.208013+0000 mon.a (mon.0) 856 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:54.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:54 smithi116 bash[65980]: audit 2023-07-10T17:15:53.209546+0000 mon.a (mon.0) 857 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.yqkmva", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:15:54.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:54 smithi116 bash[65980]: audit 2023-07-10T17:15:53.211941+0000 mon.a (mon.0) 858 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:54.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:54 smithi116 bash[65980]: cephadm 2023-07-10T17:15:53.213022+0000 mgr.y (mgr.25390) 345 : cephadm 1 Deploying daemon rgw.smpl.smithi192.yqkmva on smithi192 2023-07-10T17:15:54.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:54 smithi116 bash[63900]: cephadm 2023-07-10T17:15:53.199145+0000 mgr.y (mgr.25390) 344 : cephadm 1 Upgrade: Updating rgw.smpl.smithi192.yqkmva (4/4) 2023-07-10T17:15:54.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:54 smithi116 bash[63900]: audit 2023-07-10T17:15:53.208013+0000 mon.a (mon.0) 856 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:54.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:54 smithi116 bash[63900]: audit 2023-07-10T17:15:53.209546+0000 mon.a (mon.0) 857 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.yqkmva", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-10T17:15:54.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:54 smithi116 bash[63900]: audit 2023-07-10T17:15:53.211941+0000 mon.a (mon.0) 858 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:15:54.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:54 smithi116 bash[63900]: cephadm 2023-07-10T17:15:53.213022+0000 mgr.y (mgr.25390) 345 : cephadm 1 Deploying daemon rgw.smpl.smithi192.yqkmva on smithi192 2023-07-10T17:15:54.902 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:15:54.902 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:15:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:15:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:15:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:15:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "osd", 2023-07-10T17:15:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2023-07-10T17:15:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr" 2023-07-10T17:15:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:15:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "13/23 daemons upgraded", 2023-07-10T17:15:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading rgw daemons", 2023-07-10T17:15:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:15:54.905 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:15:55.334 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:15:55.334 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:15:55.334 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:15:55.334 INFO:teuthology.orchestra.run.smithi116.stdout: daemon prometheus.a on smithi192 is in error state 2023-07-10T17:15:55.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:55 smithi192 bash[50362]: audit 2023-07-10T17:15:53.324299+0000 mgr.y (mgr.25390) 346 : audit 0 from='client.44662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:55.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:55 smithi192 bash[50362]: cluster 2023-07-10T17:15:53.587662+0000 mgr.y (mgr.25390) 347 : cluster 0 pgmap v250: 161 pgs: 161 active+clean; 604 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 96 B/s wr, 113 op/s 2023-07-10T17:15:55.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:55 smithi192 bash[50362]: audit 2023-07-10T17:15:53.706030+0000 mgr.y (mgr.25390) 348 : audit 0 from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:55.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:55 smithi192 bash[50362]: audit 2023-07-10T17:15:54.070676+0000 mgr.y (mgr.25390) 349 : audit 0 from='client.44674 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:55.593 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:55 smithi192 bash[50362]: audit 2023-07-10T17:15:54.525078+0000 mon.c (mon.1) 23 : audit 0 from='client.44680 172.21.15.116:0/2912259455' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:55.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:55 smithi116 bash[63900]: audit 2023-07-10T17:15:53.324299+0000 mgr.y (mgr.25390) 346 : audit 0 from='client.44662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:55.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:55 smithi116 bash[63900]: cluster 2023-07-10T17:15:53.587662+0000 mgr.y (mgr.25390) 347 : cluster 0 pgmap v250: 161 pgs: 161 active+clean; 604 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 96 B/s wr, 113 op/s 2023-07-10T17:15:55.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:55 smithi116 bash[63900]: audit 2023-07-10T17:15:53.706030+0000 mgr.y (mgr.25390) 348 : audit 0 from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:55.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:55 smithi116 bash[63900]: audit 2023-07-10T17:15:54.070676+0000 mgr.y (mgr.25390) 349 : audit 0 from='client.44674 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:55.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:55 smithi116 bash[63900]: audit 2023-07-10T17:15:54.525078+0000 mon.c (mon.1) 23 : audit 0 from='client.44680 172.21.15.116:0/2912259455' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:55.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:55 smithi116 bash[65980]: audit 2023-07-10T17:15:53.324299+0000 mgr.y (mgr.25390) 346 : audit 0 from='client.44662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:55.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:55 smithi116 bash[65980]: cluster 2023-07-10T17:15:53.587662+0000 mgr.y (mgr.25390) 347 : cluster 0 pgmap v250: 161 pgs: 161 active+clean; 604 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 96 B/s wr, 113 op/s 2023-07-10T17:15:55.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:55 smithi116 bash[65980]: audit 2023-07-10T17:15:53.706030+0000 mgr.y (mgr.25390) 348 : audit 0 from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:55.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:55 smithi116 bash[65980]: audit 2023-07-10T17:15:54.070676+0000 mgr.y (mgr.25390) 349 : audit 0 from='client.44674 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:55.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:55 smithi116 bash[65980]: audit 2023-07-10T17:15:54.525078+0000 mon.c (mon.1) 23 : audit 0 from='client.44680 172.21.15.116:0/2912259455' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:15:56.633 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:56 smithi192 bash[50362]: audit 2023-07-10T17:15:54.896401+0000 mgr.y (mgr.25390) 350 : audit 0 from='client.34830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:56.633 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:56 smithi192 bash[50362]: audit 2023-07-10T17:15:55.329266+0000 mon.a (mon.0) 859 : audit 0 from='client.34836 172.21.15.116:0/4220904216' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:15:56.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:56 smithi116 bash[63900]: audit 2023-07-10T17:15:54.896401+0000 mgr.y (mgr.25390) 350 : audit 0 from='client.34830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:56.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:56 smithi116 bash[63900]: audit 2023-07-10T17:15:55.329266+0000 mon.a (mon.0) 859 : audit 0 from='client.34836 172.21.15.116:0/4220904216' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:15:56.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:56 smithi116 bash[65980]: audit 2023-07-10T17:15:54.896401+0000 mgr.y (mgr.25390) 350 : audit 0 from='client.34830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:15:56.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:56 smithi116 bash[65980]: audit 2023-07-10T17:15:55.329266+0000 mon.a (mon.0) 859 : audit 0 from='client.34836 172.21.15.116:0/4220904216' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:15:57.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:57 smithi192 bash[50362]: cluster 2023-07-10T17:15:55.588734+0000 mgr.y (mgr.25390) 351 : cluster 0 pgmap v251: 161 pgs: 161 active+clean; 604 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 85 B/s wr, 126 op/s 2023-07-10T17:15:57.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:57 smithi192 bash[50362]: audit 2023-07-10T17:15:57.017605+0000 mon.a (mon.0) 860 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:57.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:57 smithi192 bash[50362]: audit 2023-07-10T17:15:57.029077+0000 mon.a (mon.0) 861 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:57.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:57 smithi116 bash[63900]: cluster 2023-07-10T17:15:55.588734+0000 mgr.y (mgr.25390) 351 : cluster 0 pgmap v251: 161 pgs: 161 active+clean; 604 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 85 B/s wr, 126 op/s 2023-07-10T17:15:57.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:57 smithi116 bash[63900]: audit 2023-07-10T17:15:57.017605+0000 mon.a (mon.0) 860 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:57.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:57 smithi116 bash[63900]: audit 2023-07-10T17:15:57.029077+0000 mon.a (mon.0) 861 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:57.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:57 smithi116 bash[65980]: cluster 2023-07-10T17:15:55.588734+0000 mgr.y (mgr.25390) 351 : cluster 0 pgmap v251: 161 pgs: 161 active+clean; 604 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 85 B/s wr, 126 op/s 2023-07-10T17:15:57.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:57 smithi116 bash[65980]: audit 2023-07-10T17:15:57.017605+0000 mon.a (mon.0) 860 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:57.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:57 smithi116 bash[65980]: audit 2023-07-10T17:15:57.029077+0000 mon.a (mon.0) 861 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:15:59.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:15:59 smithi192 bash[50362]: cluster 2023-07-10T17:15:57.590299+0000 mgr.y (mgr.25390) 352 : cluster 0 pgmap v252: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2023-07-10T17:15:59.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:15:59 smithi116 bash[63900]: cluster 2023-07-10T17:15:57.590299+0000 mgr.y (mgr.25390) 352 : cluster 0 pgmap v252: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2023-07-10T17:15:59.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:15:59 smithi116 bash[65980]: cluster 2023-07-10T17:15:57.590299+0000 mgr.y (mgr.25390) 352 : cluster 0 pgmap v252: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2023-07-10T17:16:01.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:01 smithi192 bash[50362]: cluster 2023-07-10T17:15:59.591353+0000 mgr.y (mgr.25390) 353 : cluster 0 pgmap v253: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 85 B/s wr, 128 op/s 2023-07-10T17:16:01.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:01 smithi116 bash[63900]: cluster 2023-07-10T17:15:59.591353+0000 mgr.y (mgr.25390) 353 : cluster 0 pgmap v253: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 85 B/s wr, 128 op/s 2023-07-10T17:16:01.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:01 smithi116 bash[65980]: cluster 2023-07-10T17:15:59.591353+0000 mgr.y (mgr.25390) 353 : cluster 0 pgmap v253: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 85 B/s wr, 128 op/s 2023-07-10T17:16:03.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:03 smithi192 bash[50362]: cluster 2023-07-10T17:16:01.592721+0000 mgr.y (mgr.25390) 354 : cluster 0 pgmap v254: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 170 B/s wr, 160 op/s 2023-07-10T17:16:03.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:03 smithi192 bash[50362]: audit 2023-07-10T17:16:03.131117+0000 mon.a (mon.0) 862 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:03.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:03 smithi192 bash[50362]: audit 2023-07-10T17:16:03.140357+0000 mon.a (mon.0) 863 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:03.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:03 smithi192 bash[50362]: audit 2023-07-10T17:16:03.184926+0000 mon.a (mon.0) 864 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:03.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:03 smithi192 bash[50362]: audit 2023-07-10T17:16:03.193860+0000 mon.a (mon.0) 865 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:03.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:03 smithi116 bash[65980]: cluster 2023-07-10T17:16:01.592721+0000 mgr.y (mgr.25390) 354 : cluster 0 pgmap v254: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 170 B/s wr, 160 op/s 2023-07-10T17:16:03.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:03 smithi116 bash[65980]: audit 2023-07-10T17:16:03.131117+0000 mon.a (mon.0) 862 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:03.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:03 smithi116 bash[65980]: audit 2023-07-10T17:16:03.140357+0000 mon.a (mon.0) 863 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:03.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:03 smithi116 bash[65980]: audit 2023-07-10T17:16:03.184926+0000 mon.a (mon.0) 864 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:03.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:03 smithi116 bash[65980]: audit 2023-07-10T17:16:03.193860+0000 mon.a (mon.0) 865 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:03.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:03 smithi116 bash[63900]: cluster 2023-07-10T17:16:01.592721+0000 mgr.y (mgr.25390) 354 : cluster 0 pgmap v254: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 170 B/s wr, 160 op/s 2023-07-10T17:16:03.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:03 smithi116 bash[63900]: audit 2023-07-10T17:16:03.131117+0000 mon.a (mon.0) 862 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:03.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:03 smithi116 bash[63900]: audit 2023-07-10T17:16:03.140357+0000 mon.a (mon.0) 863 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:03.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:03 smithi116 bash[63900]: audit 2023-07-10T17:16:03.184926+0000 mon.a (mon.0) 864 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:03.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:03 smithi116 bash[63900]: audit 2023-07-10T17:16:03.193860+0000 mon.a (mon.0) 865 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:05.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:04 smithi192 bash[50362]: cluster 2023-07-10T17:16:03.593504+0000 mgr.y (mgr.25390) 355 : cluster 0 pgmap v255: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2023-07-10T17:16:05.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:04 smithi192 bash[50362]: audit 2023-07-10T17:16:03.997854+0000 mon.a (mon.0) 866 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:05.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:04 smithi192 bash[50362]: audit 2023-07-10T17:16:04.002278+0000 mon.a (mon.0) 867 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:05.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:04 smithi192 bash[50362]: audit 2023-07-10T17:16:04.012228+0000 mon.a (mon.0) 868 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:05.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:04 smithi192 bash[50362]: audit 2023-07-10T17:16:04.020163+0000 mon.a (mon.0) 869 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:05.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:04 smithi116 bash[63900]: cluster 2023-07-10T17:16:03.593504+0000 mgr.y (mgr.25390) 355 : cluster 0 pgmap v255: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2023-07-10T17:16:05.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:04 smithi116 bash[63900]: audit 2023-07-10T17:16:03.997854+0000 mon.a (mon.0) 866 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:05.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:04 smithi116 bash[63900]: audit 2023-07-10T17:16:04.002278+0000 mon.a (mon.0) 867 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:05.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:04 smithi116 bash[63900]: audit 2023-07-10T17:16:04.012228+0000 mon.a (mon.0) 868 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:05.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:04 smithi116 bash[63900]: audit 2023-07-10T17:16:04.020163+0000 mon.a (mon.0) 869 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:05.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:04 smithi116 bash[65980]: cluster 2023-07-10T17:16:03.593504+0000 mgr.y (mgr.25390) 355 : cluster 0 pgmap v255: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2023-07-10T17:16:05.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:04 smithi116 bash[65980]: audit 2023-07-10T17:16:03.997854+0000 mon.a (mon.0) 866 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:05.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:04 smithi116 bash[65980]: audit 2023-07-10T17:16:04.002278+0000 mon.a (mon.0) 867 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:05.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:04 smithi116 bash[65980]: audit 2023-07-10T17:16:04.012228+0000 mon.a (mon.0) 868 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:05.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:04 smithi116 bash[65980]: audit 2023-07-10T17:16:04.020163+0000 mon.a (mon.0) 869 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:07.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:07 smithi192 bash[50362]: cluster 2023-07-10T17:16:05.594591+0000 mgr.y (mgr.25390) 356 : cluster 0 pgmap v256: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2023-07-10T17:16:07.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:07 smithi116 bash[63900]: cluster 2023-07-10T17:16:05.594591+0000 mgr.y (mgr.25390) 356 : cluster 0 pgmap v256: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2023-07-10T17:16:07.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:07 smithi116 bash[65980]: cluster 2023-07-10T17:16:05.594591+0000 mgr.y (mgr.25390) 356 : cluster 0 pgmap v256: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2023-07-10T17:16:09.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:09 smithi192 bash[50362]: cluster 2023-07-10T17:16:07.596165+0000 mgr.y (mgr.25390) 357 : cluster 0 pgmap v257: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 170 B/s wr, 139 op/s 2023-07-10T17:16:09.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:09 smithi116 bash[63900]: cluster 2023-07-10T17:16:07.596165+0000 mgr.y (mgr.25390) 357 : cluster 0 pgmap v257: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 170 B/s wr, 139 op/s 2023-07-10T17:16:09.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:09 smithi116 bash[65980]: cluster 2023-07-10T17:16:07.596165+0000 mgr.y (mgr.25390) 357 : cluster 0 pgmap v257: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 170 B/s wr, 139 op/s 2023-07-10T17:16:11.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:11 smithi192 bash[50362]: cluster 2023-07-10T17:16:09.597084+0000 mgr.y (mgr.25390) 358 : cluster 0 pgmap v258: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 85 B/s wr, 82 op/s 2023-07-10T17:16:11.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:11 smithi116 bash[63900]: cluster 2023-07-10T17:16:09.597084+0000 mgr.y (mgr.25390) 358 : cluster 0 pgmap v258: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 85 B/s wr, 82 op/s 2023-07-10T17:16:11.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:11 smithi116 bash[65980]: cluster 2023-07-10T17:16:09.597084+0000 mgr.y (mgr.25390) 358 : cluster 0 pgmap v258: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 85 B/s wr, 82 op/s 2023-07-10T17:16:13.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:13 smithi192 bash[50362]: cluster 2023-07-10T17:16:11.598412+0000 mgr.y (mgr.25390) 359 : cluster 0 pgmap v259: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 85 B/s wr, 72 op/s 2023-07-10T17:16:13.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:13 smithi116 bash[63900]: cluster 2023-07-10T17:16:11.598412+0000 mgr.y (mgr.25390) 359 : cluster 0 pgmap v259: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 85 B/s wr, 72 op/s 2023-07-10T17:16:13.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:13 smithi116 bash[65980]: cluster 2023-07-10T17:16:11.598412+0000 mgr.y (mgr.25390) 359 : cluster 0 pgmap v259: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 85 B/s wr, 72 op/s 2023-07-10T17:16:14.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: cephadm 2023-07-10T17:16:13.514154+0000 mgr.y (mgr.25390) 360 : cephadm 1 Detected new or changed devices on smithi192 2023-07-10T17:16:14.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.525050+0000 mon.a (mon.0) 870 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.535776+0000 mon.a (mon.0) 871 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: cluster 2023-07-10T17:16:13.599634+0000 mgr.y (mgr.25390) 361 : cluster 0 pgmap v260: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 29 op/s 2023-07-10T17:16:14.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: cephadm 2023-07-10T17:16:13.702558+0000 mgr.y (mgr.25390) 362 : cephadm 1 Detected new or changed devices on smithi116 2023-07-10T17:16:14.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.711417+0000 mon.a (mon.0) 872 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.721419+0000 mon.a (mon.0) 873 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.723201+0000 mon.a (mon.0) 874 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:16:14.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.724783+0000 mon.a (mon.0) 875 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:16:14.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: cephadm 2023-07-10T17:16:13.726101+0000 mgr.y (mgr.25390) 363 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:16:14.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: cluster 2023-07-10T17:16:13.727258+0000 mgr.y (mgr.25390) 364 : cluster 0 pgmap v261: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:14.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: cluster 2023-07-10T17:16:13.727648+0000 mgr.y (mgr.25390) 365 : cluster 0 pgmap v262: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:14.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.735153+0000 mon.a (mon.0) 876 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.825007+0000 mon.a (mon.0) 877 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:16:14.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.829913+0000 mon.a (mon.0) 878 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.832680+0000 mon.a (mon.0) 879 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.834792+0000 mon.a (mon.0) 880 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.838280+0000 mon.a (mon.0) 881 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.841976+0000 mon.a (mon.0) 882 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.650 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.844749+0000 mon.a (mon.0) 883 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: cephadm 2023-07-10T17:16:13.514154+0000 mgr.y (mgr.25390) 360 : cephadm 1 Detected new or changed devices on smithi192 2023-07-10T17:16:14.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.525050+0000 mon.a (mon.0) 870 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.535776+0000 mon.a (mon.0) 871 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: cluster 2023-07-10T17:16:13.599634+0000 mgr.y (mgr.25390) 361 : cluster 0 pgmap v260: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 29 op/s 2023-07-10T17:16:14.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: cephadm 2023-07-10T17:16:13.702558+0000 mgr.y (mgr.25390) 362 : cephadm 1 Detected new or changed devices on smithi116 2023-07-10T17:16:14.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.711417+0000 mon.a (mon.0) 872 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.721419+0000 mon.a (mon.0) 873 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.723201+0000 mon.a (mon.0) 874 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:16:14.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.724783+0000 mon.a (mon.0) 875 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:16:14.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: cephadm 2023-07-10T17:16:13.726101+0000 mgr.y (mgr.25390) 363 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:16:14.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: cluster 2023-07-10T17:16:13.727258+0000 mgr.y (mgr.25390) 364 : cluster 0 pgmap v261: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:14.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: cluster 2023-07-10T17:16:13.727648+0000 mgr.y (mgr.25390) 365 : cluster 0 pgmap v262: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:14.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.735153+0000 mon.a (mon.0) 876 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.652 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.825007+0000 mon.a (mon.0) 877 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:16:14.653 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.829913+0000 mon.a (mon.0) 878 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.653 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.832680+0000 mon.a (mon.0) 879 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.653 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.834792+0000 mon.a (mon.0) 880 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.653 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.838280+0000 mon.a (mon.0) 881 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.653 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.841976+0000 mon.a (mon.0) 882 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.653 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.844749+0000 mon.a (mon.0) 883 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: cephadm 2023-07-10T17:16:13.846123+0000 mgr.y (mgr.25390) 366 : cephadm 1 Upgrade: Setting container_image for all rgw 2023-07-10T17:16:14.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.852777+0000 mon.a (mon.0) 884 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.854547+0000 mon.a (mon.0) 885 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi116.stnxgn"}]: dispatch 2023-07-10T17:16:14.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.861226+0000 mon.a (mon.0) 886 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi116.stnxgn"}]': finished 2023-07-10T17:16:14.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.863334+0000 mon.a (mon.0) 887 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi192.myrhsb"}]: dispatch 2023-07-10T17:16:14.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.872299+0000 mon.a (mon.0) 888 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi192.myrhsb"}]': finished 2023-07-10T17:16:14.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.873684+0000 mon.a (mon.0) 889 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi116.vpoihf"}]: dispatch 2023-07-10T17:16:14.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.880222+0000 mon.a (mon.0) 890 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi116.vpoihf"}]': finished 2023-07-10T17:16:14.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.881324+0000 mon.a (mon.0) 891 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi192.yqkmva"}]: dispatch 2023-07-10T17:16:14.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.888026+0000 mon.a (mon.0) 892 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi192.yqkmva"}]': finished 2023-07-10T17:16:14.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.890135+0000 mon.a (mon.0) 893 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: cephadm 2023-07-10T17:16:13.891408+0000 mgr.y (mgr.25390) 367 : cephadm 1 Upgrade: Setting container_image for all rbd-mirror 2023-07-10T17:16:14.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.897644+0000 mon.a (mon.0) 894 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.899902+0000 mon.a (mon.0) 895 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.902016+0000 mon.a (mon.0) 896 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: cephadm 2023-07-10T17:16:13.903114+0000 mgr.y (mgr.25390) 368 : cephadm 1 Upgrade: Setting container_image for all ceph-exporter 2023-07-10T17:16:14.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[63900]: audit 2023-07-10T17:16:13.910439+0000 mon.a (mon.0) 897 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.658 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: cephadm 2023-07-10T17:16:13.846123+0000 mgr.y (mgr.25390) 366 : cephadm 1 Upgrade: Setting container_image for all rgw 2023-07-10T17:16:14.658 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.852777+0000 mon.a (mon.0) 884 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.658 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.854547+0000 mon.a (mon.0) 885 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi116.stnxgn"}]: dispatch 2023-07-10T17:16:14.659 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.861226+0000 mon.a (mon.0) 886 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi116.stnxgn"}]': finished 2023-07-10T17:16:14.659 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.863334+0000 mon.a (mon.0) 887 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi192.myrhsb"}]: dispatch 2023-07-10T17:16:14.659 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.872299+0000 mon.a (mon.0) 888 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi192.myrhsb"}]': finished 2023-07-10T17:16:14.659 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.873684+0000 mon.a (mon.0) 889 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi116.vpoihf"}]: dispatch 2023-07-10T17:16:14.659 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.880222+0000 mon.a (mon.0) 890 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi116.vpoihf"}]': finished 2023-07-10T17:16:14.659 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.881324+0000 mon.a (mon.0) 891 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi192.yqkmva"}]: dispatch 2023-07-10T17:16:14.660 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.888026+0000 mon.a (mon.0) 892 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi192.yqkmva"}]': finished 2023-07-10T17:16:14.660 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.890135+0000 mon.a (mon.0) 893 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.660 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: cephadm 2023-07-10T17:16:13.891408+0000 mgr.y (mgr.25390) 367 : cephadm 1 Upgrade: Setting container_image for all rbd-mirror 2023-07-10T17:16:14.661 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.897644+0000 mon.a (mon.0) 894 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.661 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.899902+0000 mon.a (mon.0) 895 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.661 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.902016+0000 mon.a (mon.0) 896 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.662 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: cephadm 2023-07-10T17:16:13.903114+0000 mgr.y (mgr.25390) 368 : cephadm 1 Upgrade: Setting container_image for all ceph-exporter 2023-07-10T17:16:14.662 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:14 smithi116 bash[65980]: audit 2023-07-10T17:16:13.910439+0000 mon.a (mon.0) 897 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: cephadm 2023-07-10T17:16:13.514154+0000 mgr.y (mgr.25390) 360 : cephadm 1 Detected new or changed devices on smithi192 2023-07-10T17:16:14.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.525050+0000 mon.a (mon.0) 870 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.535776+0000 mon.a (mon.0) 871 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: cluster 2023-07-10T17:16:13.599634+0000 mgr.y (mgr.25390) 361 : cluster 0 pgmap v260: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 29 op/s 2023-07-10T17:16:14.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: cephadm 2023-07-10T17:16:13.702558+0000 mgr.y (mgr.25390) 362 : cephadm 1 Detected new or changed devices on smithi116 2023-07-10T17:16:14.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.711417+0000 mon.a (mon.0) 872 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.721419+0000 mon.a (mon.0) 873 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.723201+0000 mon.a (mon.0) 874 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:16:14.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.724783+0000 mon.a (mon.0) 875 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:16:14.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: cephadm 2023-07-10T17:16:13.726101+0000 mgr.y (mgr.25390) 363 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:16:14.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: cluster 2023-07-10T17:16:13.727258+0000 mgr.y (mgr.25390) 364 : cluster 0 pgmap v261: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:14.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: cluster 2023-07-10T17:16:13.727648+0000 mgr.y (mgr.25390) 365 : cluster 0 pgmap v262: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:14.892 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.735153+0000 mon.a (mon.0) 876 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.892 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.825007+0000 mon.a (mon.0) 877 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:16:14.893 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.829913+0000 mon.a (mon.0) 878 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.893 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.832680+0000 mon.a (mon.0) 879 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.893 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.834792+0000 mon.a (mon.0) 880 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.893 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.838280+0000 mon.a (mon.0) 881 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.894 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.841976+0000 mon.a (mon.0) 882 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.894 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.844749+0000 mon.a (mon.0) 883 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.894 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: cephadm 2023-07-10T17:16:13.846123+0000 mgr.y (mgr.25390) 366 : cephadm 1 Upgrade: Setting container_image for all rgw 2023-07-10T17:16:14.895 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.852777+0000 mon.a (mon.0) 884 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.896 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.854547+0000 mon.a (mon.0) 885 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi116.stnxgn"}]: dispatch 2023-07-10T17:16:14.896 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.861226+0000 mon.a (mon.0) 886 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi116.stnxgn"}]': finished 2023-07-10T17:16:14.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.863334+0000 mon.a (mon.0) 887 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi192.myrhsb"}]: dispatch 2023-07-10T17:16:14.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.872299+0000 mon.a (mon.0) 888 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi192.myrhsb"}]': finished 2023-07-10T17:16:14.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.873684+0000 mon.a (mon.0) 889 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi116.vpoihf"}]: dispatch 2023-07-10T17:16:14.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.880222+0000 mon.a (mon.0) 890 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi116.vpoihf"}]': finished 2023-07-10T17:16:14.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.881324+0000 mon.a (mon.0) 891 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi192.yqkmva"}]: dispatch 2023-07-10T17:16:14.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.888026+0000 mon.a (mon.0) 892 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi192.yqkmva"}]': finished 2023-07-10T17:16:14.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.890135+0000 mon.a (mon.0) 893 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.898 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: cephadm 2023-07-10T17:16:13.891408+0000 mgr.y (mgr.25390) 367 : cephadm 1 Upgrade: Setting container_image for all rbd-mirror 2023-07-10T17:16:14.898 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.897644+0000 mon.a (mon.0) 894 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:14.898 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.899902+0000 mon.a (mon.0) 895 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.898 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.902016+0000 mon.a (mon.0) 896 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:14.898 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: cephadm 2023-07-10T17:16:13.903114+0000 mgr.y (mgr.25390) 368 : cephadm 1 Upgrade: Setting container_image for all ceph-exporter 2023-07-10T17:16:14.898 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:14 smithi192 bash[50362]: audit 2023-07-10T17:16:13.910439+0000 mon.a (mon.0) 897 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:17.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:16 smithi192 bash[50362]: cluster 2023-07-10T17:16:15.728755+0000 mgr.y (mgr.25390) 369 : cluster 0 pgmap v263: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:17.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:16 smithi192 bash[50362]: audit 2023-07-10T17:16:16.342298+0000 mon.a (mon.0) 898 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:17.284 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:16 smithi192 bash[50362]: audit 2023-07-10T17:16:16.343985+0000 mon.a (mon.0) 899 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:16:17.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:16 smithi192 bash[50362]: audit 2023-07-10T17:16:16.354213+0000 mon.a (mon.0) 900 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:16:17.286 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:16 smithi116 bash[65980]: cluster 2023-07-10T17:16:15.728755+0000 mgr.y (mgr.25390) 369 : cluster 0 pgmap v263: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:17.286 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:16 smithi116 bash[65980]: audit 2023-07-10T17:16:16.342298+0000 mon.a (mon.0) 898 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:17.286 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:16 smithi116 bash[65980]: audit 2023-07-10T17:16:16.343985+0000 mon.a (mon.0) 899 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:16:17.286 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:16 smithi116 bash[65980]: audit 2023-07-10T17:16:16.354213+0000 mon.a (mon.0) 900 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:16:17.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:16 smithi116 bash[63900]: cluster 2023-07-10T17:16:15.728755+0000 mgr.y (mgr.25390) 369 : cluster 0 pgmap v263: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:17.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:16 smithi116 bash[63900]: audit 2023-07-10T17:16:16.342298+0000 mon.a (mon.0) 898 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:17.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:16 smithi116 bash[63900]: audit 2023-07-10T17:16:16.343985+0000 mon.a (mon.0) 899 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi116.fkthyl", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-10T17:16:17.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:16 smithi116 bash[63900]: audit 2023-07-10T17:16:16.354213+0000 mon.a (mon.0) 900 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:16:18.044 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:17 smithi116 bash[63900]: cephadm 2023-07-10T17:16:16.332222+0000 mgr.y (mgr.25390) 370 : cephadm 1 Upgrade: Updating iscsi.foo.smithi116.fkthyl 2023-07-10T17:16:18.044 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:17 smithi116 bash[63900]: cephadm 2023-07-10T17:16:16.355865+0000 mgr.y (mgr.25390) 371 : cephadm 1 Deploying daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:16:18.044 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:17 smithi116 bash[65980]: cephadm 2023-07-10T17:16:16.332222+0000 mgr.y (mgr.25390) 370 : cephadm 1 Upgrade: Updating iscsi.foo.smithi116.fkthyl 2023-07-10T17:16:18.044 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:17 smithi116 bash[65980]: cephadm 2023-07-10T17:16:16.355865+0000 mgr.y (mgr.25390) 371 : cephadm 1 Deploying daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:16:18.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:17 smithi192 bash[50362]: cephadm 2023-07-10T17:16:16.332222+0000 mgr.y (mgr.25390) 370 : cephadm 1 Upgrade: Updating iscsi.foo.smithi116.fkthyl 2023-07-10T17:16:18.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:17 smithi192 bash[50362]: cephadm 2023-07-10T17:16:16.355865+0000 mgr.y (mgr.25390) 371 : cephadm 1 Deploying daemon iscsi.foo.smithi116.fkthyl on smithi116 2023-07-10T17:16:19.109 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:18 smithi116 bash[63900]: cluster 2023-07-10T17:16:17.730277+0000 mgr.y (mgr.25390) 372 : cluster 0 pgmap v264: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:19.110 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:18 smithi116 bash[65980]: cluster 2023-07-10T17:16:17.730277+0000 mgr.y (mgr.25390) 372 : cluster 0 pgmap v264: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:19.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:18 smithi192 bash[50362]: cluster 2023-07-10T17:16:17.730277+0000 mgr.y (mgr.25390) 372 : cluster 0 pgmap v264: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:20.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:20 smithi192 bash[50362]: audit 2023-07-10T17:16:19.153193+0000 mon.a (mon.0) 901 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:20.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:20 smithi192 bash[50362]: audit 2023-07-10T17:16:19.160110+0000 mon.a (mon.0) 902 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:20.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:20 smithi192 bash[50362]: audit 2023-07-10T17:16:19.551049+0000 mon.a (mon.0) 903 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:20.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:20 smithi192 bash[50362]: audit 2023-07-10T17:16:19.561082+0000 mon.a (mon.0) 904 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:20.680 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:20 smithi116 bash[65980]: audit 2023-07-10T17:16:19.153193+0000 mon.a (mon.0) 901 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:20.680 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:20 smithi116 bash[65980]: audit 2023-07-10T17:16:19.160110+0000 mon.a (mon.0) 902 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:20.681 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:20 smithi116 bash[65980]: audit 2023-07-10T17:16:19.551049+0000 mon.a (mon.0) 903 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:20.681 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:20 smithi116 bash[65980]: audit 2023-07-10T17:16:19.561082+0000 mon.a (mon.0) 904 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:20.681 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:20 smithi116 bash[63900]: audit 2023-07-10T17:16:19.153193+0000 mon.a (mon.0) 901 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:20.681 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:20 smithi116 bash[63900]: audit 2023-07-10T17:16:19.160110+0000 mon.a (mon.0) 902 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:20.681 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:20 smithi116 bash[63900]: audit 2023-07-10T17:16:19.551049+0000 mon.a (mon.0) 903 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:20.681 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:20 smithi116 bash[63900]: audit 2023-07-10T17:16:19.561082+0000 mon.a (mon.0) 904 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:21.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:21 smithi116 bash[65980]: cluster 2023-07-10T17:16:19.731356+0000 mgr.y (mgr.25390) 373 : cluster 0 pgmap v265: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:21.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:21 smithi116 bash[63900]: cluster 2023-07-10T17:16:19.731356+0000 mgr.y (mgr.25390) 373 : cluster 0 pgmap v265: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:21.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:21 smithi192 bash[50362]: cluster 2023-07-10T17:16:19.731356+0000 mgr.y (mgr.25390) 373 : cluster 0 pgmap v265: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:22.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:22 smithi192 bash[50362]: audit 2023-07-10T17:16:21.518251+0000 mon.c (mon.1) 24 : audit 0 from='client.? 172.21.15.116:0/2942549477' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-10T17:16:22.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:22 smithi192 bash[50362]: audit 2023-07-10T17:16:21.895162+0000 mon.a (mon.0) 905 : audit 1 from='client.? 172.21.15.116:0/533280892' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/1249272063"}]: dispatch 2023-07-10T17:16:22.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:22 smithi116 bash[63900]: audit 2023-07-10T17:16:21.518251+0000 mon.c (mon.1) 24 : audit 0 from='client.? 172.21.15.116:0/2942549477' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-10T17:16:22.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:22 smithi116 bash[63900]: audit 2023-07-10T17:16:21.895162+0000 mon.a (mon.0) 905 : audit 1 from='client.? 172.21.15.116:0/533280892' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/1249272063"}]: dispatch 2023-07-10T17:16:22.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:22 smithi116 bash[65980]: audit 2023-07-10T17:16:21.518251+0000 mon.c (mon.1) 24 : audit 0 from='client.? 172.21.15.116:0/2942549477' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-10T17:16:22.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:22 smithi116 bash[65980]: audit 2023-07-10T17:16:21.895162+0000 mon.a (mon.0) 905 : audit 1 from='client.? 172.21.15.116:0/533280892' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/1249272063"}]: dispatch 2023-07-10T17:16:23.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:23 smithi192 bash[50362]: cluster 2023-07-10T17:16:21.732771+0000 mgr.y (mgr.25390) 374 : cluster 0 pgmap v266: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:23.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:23 smithi192 bash[50362]: audit 2023-07-10T17:16:22.171675+0000 mon.a (mon.0) 906 : audit 1 from='client.? 172.21.15.116:0/533280892' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/1249272063"}]': finished 2023-07-10T17:16:23.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:23 smithi192 bash[50362]: cluster 2023-07-10T17:16:22.171764+0000 mon.a (mon.0) 907 : cluster 0 osdmap e124: 8 total, 8 up, 8 in 2023-07-10T17:16:23.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:23 smithi192 bash[50362]: audit 2023-07-10T17:16:22.559101+0000 mon.a (mon.0) 908 : audit 1 from='client.? 172.21.15.116:0/4160295217' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]: dispatch 2023-07-10T17:16:23.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:23 smithi116 bash[65980]: cluster 2023-07-10T17:16:21.732771+0000 mgr.y (mgr.25390) 374 : cluster 0 pgmap v266: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:23.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:23 smithi116 bash[65980]: audit 2023-07-10T17:16:22.171675+0000 mon.a (mon.0) 906 : audit 1 from='client.? 172.21.15.116:0/533280892' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/1249272063"}]': finished 2023-07-10T17:16:23.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:23 smithi116 bash[65980]: cluster 2023-07-10T17:16:22.171764+0000 mon.a (mon.0) 907 : cluster 0 osdmap e124: 8 total, 8 up, 8 in 2023-07-10T17:16:23.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:23 smithi116 bash[65980]: audit 2023-07-10T17:16:22.559101+0000 mon.a (mon.0) 908 : audit 1 from='client.? 172.21.15.116:0/4160295217' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]: dispatch 2023-07-10T17:16:23.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:23 smithi116 bash[63900]: cluster 2023-07-10T17:16:21.732771+0000 mgr.y (mgr.25390) 374 : cluster 0 pgmap v266: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:23.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:23 smithi116 bash[63900]: audit 2023-07-10T17:16:22.171675+0000 mon.a (mon.0) 906 : audit 1 from='client.? 172.21.15.116:0/533280892' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/1249272063"}]': finished 2023-07-10T17:16:23.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:23 smithi116 bash[63900]: cluster 2023-07-10T17:16:22.171764+0000 mon.a (mon.0) 907 : cluster 0 osdmap e124: 8 total, 8 up, 8 in 2023-07-10T17:16:23.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:23 smithi116 bash[63900]: audit 2023-07-10T17:16:22.559101+0000 mon.a (mon.0) 908 : audit 1 from='client.? 172.21.15.116:0/4160295217' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]: dispatch 2023-07-10T17:16:24.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:24 smithi192 bash[50362]: audit 2023-07-10T17:16:23.177025+0000 mon.a (mon.0) 909 : audit 1 from='client.? 172.21.15.116:0/4160295217' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]': finished 2023-07-10T17:16:24.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:24 smithi192 bash[50362]: cluster 2023-07-10T17:16:23.177116+0000 mon.a (mon.0) 910 : cluster 0 osdmap e125: 8 total, 8 up, 8 in 2023-07-10T17:16:24.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:24 smithi192 bash[50362]: audit 2023-07-10T17:16:23.581722+0000 mon.a (mon.0) 911 : audit 1 from='client.? 172.21.15.116:0/2964252203' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/2444686048"}]: dispatch 2023-07-10T17:16:24.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:24 smithi116 bash[65980]: audit 2023-07-10T17:16:23.177025+0000 mon.a (mon.0) 909 : audit 1 from='client.? 172.21.15.116:0/4160295217' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]': finished 2023-07-10T17:16:24.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:24 smithi116 bash[65980]: cluster 2023-07-10T17:16:23.177116+0000 mon.a (mon.0) 910 : cluster 0 osdmap e125: 8 total, 8 up, 8 in 2023-07-10T17:16:24.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:24 smithi116 bash[65980]: audit 2023-07-10T17:16:23.581722+0000 mon.a (mon.0) 911 : audit 1 from='client.? 172.21.15.116:0/2964252203' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/2444686048"}]: dispatch 2023-07-10T17:16:24.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:24 smithi116 bash[63900]: audit 2023-07-10T17:16:23.177025+0000 mon.a (mon.0) 909 : audit 1 from='client.? 172.21.15.116:0/4160295217' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1335287284"}]': finished 2023-07-10T17:16:24.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:24 smithi116 bash[63900]: cluster 2023-07-10T17:16:23.177116+0000 mon.a (mon.0) 910 : cluster 0 osdmap e125: 8 total, 8 up, 8 in 2023-07-10T17:16:24.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:24 smithi116 bash[63900]: audit 2023-07-10T17:16:23.581722+0000 mon.a (mon.0) 911 : audit 1 from='client.? 172.21.15.116:0/2964252203' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/2444686048"}]: dispatch 2023-07-10T17:16:25.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:25 smithi192 bash[50362]: cluster 2023-07-10T17:16:23.733522+0000 mgr.y (mgr.25390) 375 : cluster 0 pgmap v269: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:25.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:25 smithi192 bash[50362]: audit 2023-07-10T17:16:24.187778+0000 mon.a (mon.0) 912 : audit 1 from='client.? 172.21.15.116:0/2964252203' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/2444686048"}]': finished 2023-07-10T17:16:25.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:25 smithi192 bash[50362]: cluster 2023-07-10T17:16:24.187842+0000 mon.a (mon.0) 913 : cluster 0 osdmap e126: 8 total, 8 up, 8 in 2023-07-10T17:16:25.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:25 smithi192 bash[50362]: audit 2023-07-10T17:16:24.575915+0000 mon.c (mon.1) 25 : audit 1 from='client.? 172.21.15.116:0/476862264' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: dispatch 2023-07-10T17:16:25.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:25 smithi192 bash[50362]: audit 2023-07-10T17:16:24.576528+0000 mon.a (mon.0) 914 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: dispatch 2023-07-10T17:16:25.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:25 smithi116 bash[65980]: cluster 2023-07-10T17:16:23.733522+0000 mgr.y (mgr.25390) 375 : cluster 0 pgmap v269: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:25.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:25 smithi116 bash[65980]: audit 2023-07-10T17:16:24.187778+0000 mon.a (mon.0) 912 : audit 1 from='client.? 172.21.15.116:0/2964252203' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/2444686048"}]': finished 2023-07-10T17:16:25.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:25 smithi116 bash[65980]: cluster 2023-07-10T17:16:24.187842+0000 mon.a (mon.0) 913 : cluster 0 osdmap e126: 8 total, 8 up, 8 in 2023-07-10T17:16:25.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:25 smithi116 bash[65980]: audit 2023-07-10T17:16:24.575915+0000 mon.c (mon.1) 25 : audit 1 from='client.? 172.21.15.116:0/476862264' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: dispatch 2023-07-10T17:16:25.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:25 smithi116 bash[65980]: audit 2023-07-10T17:16:24.576528+0000 mon.a (mon.0) 914 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: dispatch 2023-07-10T17:16:25.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:25 smithi116 bash[63900]: cluster 2023-07-10T17:16:23.733522+0000 mgr.y (mgr.25390) 375 : cluster 0 pgmap v269: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:25.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:25 smithi116 bash[63900]: audit 2023-07-10T17:16:24.187778+0000 mon.a (mon.0) 912 : audit 1 from='client.? 172.21.15.116:0/2964252203' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/2444686048"}]': finished 2023-07-10T17:16:25.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:25 smithi116 bash[63900]: cluster 2023-07-10T17:16:24.187842+0000 mon.a (mon.0) 913 : cluster 0 osdmap e126: 8 total, 8 up, 8 in 2023-07-10T17:16:25.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:25 smithi116 bash[63900]: audit 2023-07-10T17:16:24.575915+0000 mon.c (mon.1) 25 : audit 1 from='client.? 172.21.15.116:0/476862264' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: dispatch 2023-07-10T17:16:25.649 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:25 smithi116 bash[63900]: audit 2023-07-10T17:16:24.576528+0000 mon.a (mon.0) 914 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]: dispatch 2023-07-10T17:16:25.768 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:16:26.559 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:16:26.560 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (9m) 0s ago 15m 10.9M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:16:26.560 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (8m) 0s ago 15m 26.3M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:16:26.560 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 running (7s) 0s ago 13m 68.8M - 3.6 17e8f0d79247 fbdecd8618b7 2023-07-10T17:16:26.560 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (7m) 0s ago 20m 430M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:16:26.561 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (8m) 0s ago 21m 481M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:16:26.561 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (6m) 0s ago 22m 80.3M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:16:26.561 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (5m) 0s ago 20m 59.9M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 74a082a2e795 2023-07-10T17:16:26.561 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (6m) 0s ago 20m 62.0M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:16:26.561 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (9m) 0s ago 16m 5971k - 0.18.1 e5a616e4b9cf 3bd058d91353 2023-07-10T17:16:26.562 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (9m) 0s ago 16m 4056k - 0.18.1 e5a616e4b9cf 63551615c679 2023-07-10T17:16:26.562 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (4m) 0s ago 19m 62.4M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 0562f587fbca 2023-07-10T17:16:26.562 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (4m) 0s ago 19m 62.9M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 5b2a12a0bfc9 2023-07-10T17:16:26.562 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (3m) 0s ago 18m 54.2M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 88096bc0faa9 2023-07-10T17:16:26.562 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (3m) 0s ago 18m 59.3M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 61ae280c471b 2023-07-10T17:16:26.563 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (2m) 0s ago 18m 59.7M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 44d809c18118 2023-07-10T17:16:26.563 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (2m) 0s ago 17m 56.0M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 179270eb7044 2023-07-10T17:16:26.563 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (100s) 0s ago 17m 55.1M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 b680bfbe9a0c 2023-07-10T17:16:26.563 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (71s) 0s ago 16m 56.1M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 bd3788803e68 2023-07-10T17:16:26.563 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 error 0s ago 16m - - 2023-07-10T17:16:26.563 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (49s) 0s ago 13m 83.7M - 18.1.2-30-g8507faa7 17e8f0d79247 ef72a4e0ae29 2023-07-10T17:16:26.563 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (36s) 0s ago 13m 83.8M - 18.1.2-30-g8507faa7 17e8f0d79247 59b0f0b41f44 2023-07-10T17:16:26.563 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (42s) 0s ago 13m 83.7M - 18.1.2-30-g8507faa7 17e8f0d79247 42f4f263b75e 2023-07-10T17:16:26.564 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (29s) 0s ago 13m 83.4M - 18.1.2-30-g8507faa7 17e8f0d79247 2a257a88c4e9 2023-07-10T17:16:26.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:26 smithi192 bash[50362]: audit 2023-07-10T17:16:25.197163+0000 mon.a (mon.0) 915 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]': finished 2023-07-10T17:16:26.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:26 smithi192 bash[50362]: cluster 2023-07-10T17:16:25.197281+0000 mon.a (mon.0) 916 : cluster 0 osdmap e127: 8 total, 8 up, 8 in 2023-07-10T17:16:26.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:26 smithi192 bash[50362]: audit 2023-07-10T17:16:25.622526+0000 mon.a (mon.0) 917 : audit 1 from='client.? 172.21.15.116:0/1236721718' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/4243441584"}]: dispatch 2023-07-10T17:16:26.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:26 smithi192 bash[50362]: audit 2023-07-10T17:16:25.656280+0000 mon.a (mon.0) 918 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:26.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:26 smithi192 bash[50362]: audit 2023-07-10T17:16:25.663567+0000 mon.a (mon.0) 919 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:26.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:26 smithi116 bash[63900]: audit 2023-07-10T17:16:25.197163+0000 mon.a (mon.0) 915 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]': finished 2023-07-10T17:16:26.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:26 smithi116 bash[63900]: cluster 2023-07-10T17:16:25.197281+0000 mon.a (mon.0) 916 : cluster 0 osdmap e127: 8 total, 8 up, 8 in 2023-07-10T17:16:26.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:26 smithi116 bash[63900]: audit 2023-07-10T17:16:25.622526+0000 mon.a (mon.0) 917 : audit 1 from='client.? 172.21.15.116:0/1236721718' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/4243441584"}]: dispatch 2023-07-10T17:16:26.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:26 smithi116 bash[63900]: audit 2023-07-10T17:16:25.656280+0000 mon.a (mon.0) 918 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:26.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:26 smithi116 bash[63900]: audit 2023-07-10T17:16:25.663567+0000 mon.a (mon.0) 919 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:26.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:26 smithi116 bash[65980]: audit 2023-07-10T17:16:25.197163+0000 mon.a (mon.0) 915 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/2647439805"}]': finished 2023-07-10T17:16:26.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:26 smithi116 bash[65980]: cluster 2023-07-10T17:16:25.197281+0000 mon.a (mon.0) 916 : cluster 0 osdmap e127: 8 total, 8 up, 8 in 2023-07-10T17:16:26.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:26 smithi116 bash[65980]: audit 2023-07-10T17:16:25.622526+0000 mon.a (mon.0) 917 : audit 1 from='client.? 172.21.15.116:0/1236721718' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/4243441584"}]: dispatch 2023-07-10T17:16:26.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:26 smithi116 bash[65980]: audit 2023-07-10T17:16:25.656280+0000 mon.a (mon.0) 918 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:26.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:26 smithi116 bash[65980]: audit 2023-07-10T17:16:25.663567+0000 mon.a (mon.0) 919 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:26.992 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:16:26.993 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:16:26.993 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:16:26.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:16:26.993 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:16:26.993 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:16:26.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:16:26.994 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:16:26.994 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 8 2023-07-10T17:16:26.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:16:26.994 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:16:26.994 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 4 2023-07-10T17:16:26.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:16:26.995 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:16:26.995 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 17 2023-07-10T17:16:26.995 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:16:26.995 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:16:27.369 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:16:27.369 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:16:27.369 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:16:27.369 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:16:27.370 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:16:27.370 INFO:teuthology.orchestra.run.smithi116.stdout: "iscsi", 2023-07-10T17:16:27.370 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw", 2023-07-10T17:16:27.370 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2023-07-10T17:16:27.370 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr", 2023-07-10T17:16:27.370 INFO:teuthology.orchestra.run.smithi116.stdout: "osd" 2023-07-10T17:16:27.371 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:16:27.371 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "18/23 daemons upgraded", 2023-07-10T17:16:27.371 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading iscsi daemons", 2023-07-10T17:16:27.371 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:16:27.371 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:16:27.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:27 smithi192 bash[50362]: cluster 2023-07-10T17:16:25.734282+0000 mgr.y (mgr.25390) 376 : cluster 0 pgmap v272: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:27.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:27 smithi192 bash[50362]: audit 2023-07-10T17:16:25.736532+0000 mgr.y (mgr.25390) 377 : audit 0 from='client.44770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:16:27.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:27 smithi192 bash[50362]: audit 2023-07-10T17:16:26.139801+0000 mgr.y (mgr.25390) 378 : audit 0 from='client.44776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:16:27.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:27 smithi192 bash[50362]: audit 2023-07-10T17:16:26.198336+0000 mon.a (mon.0) 920 : audit 1 from='client.? 172.21.15.116:0/1236721718' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/4243441584"}]': finished 2023-07-10T17:16:27.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:27 smithi192 bash[50362]: cluster 2023-07-10T17:16:26.198438+0000 mon.a (mon.0) 921 : cluster 0 osdmap e128: 8 total, 8 up, 8 in 2023-07-10T17:16:27.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:27 smithi192 bash[50362]: audit 2023-07-10T17:16:26.470960+0000 mon.a (mon.0) 922 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:27.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:27 smithi192 bash[50362]: audit 2023-07-10T17:16:26.481350+0000 mon.a (mon.0) 923 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:27.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:27 smithi192 bash[50362]: audit 2023-07-10T17:16:26.671256+0000 mon.c (mon.1) 26 : audit 1 from='client.? 172.21.15.116:0/2197914377' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/1249272063"}]: dispatch 2023-07-10T17:16:27.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:27 smithi192 bash[50362]: audit 2023-07-10T17:16:26.672354+0000 mon.a (mon.0) 924 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/1249272063"}]: dispatch 2023-07-10T17:16:27.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:27 smithi192 bash[50362]: audit 2023-07-10T17:16:26.986970+0000 mon.c (mon.1) 27 : audit 0 from='client.44794 172.21.15.116:0/3612006856' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:27.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[65980]: cluster 2023-07-10T17:16:25.734282+0000 mgr.y (mgr.25390) 376 : cluster 0 pgmap v272: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:27.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[65980]: audit 2023-07-10T17:16:25.736532+0000 mgr.y (mgr.25390) 377 : audit 0 from='client.44770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:16:27.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[65980]: audit 2023-07-10T17:16:26.139801+0000 mgr.y (mgr.25390) 378 : audit 0 from='client.44776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:16:27.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[65980]: audit 2023-07-10T17:16:26.198336+0000 mon.a (mon.0) 920 : audit 1 from='client.? 172.21.15.116:0/1236721718' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/4243441584"}]': finished 2023-07-10T17:16:27.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[65980]: cluster 2023-07-10T17:16:26.198438+0000 mon.a (mon.0) 921 : cluster 0 osdmap e128: 8 total, 8 up, 8 in 2023-07-10T17:16:27.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[65980]: audit 2023-07-10T17:16:26.470960+0000 mon.a (mon.0) 922 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:27.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[65980]: audit 2023-07-10T17:16:26.481350+0000 mon.a (mon.0) 923 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:27.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[65980]: audit 2023-07-10T17:16:26.671256+0000 mon.c (mon.1) 26 : audit 1 from='client.? 172.21.15.116:0/2197914377' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/1249272063"}]: dispatch 2023-07-10T17:16:27.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[65980]: audit 2023-07-10T17:16:26.672354+0000 mon.a (mon.0) 924 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/1249272063"}]: dispatch 2023-07-10T17:16:27.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[65980]: audit 2023-07-10T17:16:26.986970+0000 mon.c (mon.1) 27 : audit 0 from='client.44794 172.21.15.116:0/3612006856' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:27.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[63900]: cluster 2023-07-10T17:16:25.734282+0000 mgr.y (mgr.25390) 376 : cluster 0 pgmap v272: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:27.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[63900]: audit 2023-07-10T17:16:25.736532+0000 mgr.y (mgr.25390) 377 : audit 0 from='client.44770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:16:27.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[63900]: audit 2023-07-10T17:16:26.139801+0000 mgr.y (mgr.25390) 378 : audit 0 from='client.44776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:16:27.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[63900]: audit 2023-07-10T17:16:26.198336+0000 mon.a (mon.0) 920 : audit 1 from='client.? 172.21.15.116:0/1236721718' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/4243441584"}]': finished 2023-07-10T17:16:27.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[63900]: cluster 2023-07-10T17:16:26.198438+0000 mon.a (mon.0) 921 : cluster 0 osdmap e128: 8 total, 8 up, 8 in 2023-07-10T17:16:27.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[63900]: audit 2023-07-10T17:16:26.470960+0000 mon.a (mon.0) 922 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:27.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[63900]: audit 2023-07-10T17:16:26.481350+0000 mon.a (mon.0) 923 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:27.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[63900]: audit 2023-07-10T17:16:26.671256+0000 mon.c (mon.1) 26 : audit 1 from='client.? 172.21.15.116:0/2197914377' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/1249272063"}]: dispatch 2023-07-10T17:16:27.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[63900]: audit 2023-07-10T17:16:26.672354+0000 mon.a (mon.0) 924 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/1249272063"}]: dispatch 2023-07-10T17:16:27.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:27 smithi116 bash[63900]: audit 2023-07-10T17:16:26.986970+0000 mon.c (mon.1) 27 : audit 0 from='client.44794 172.21.15.116:0/3612006856' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:27.810 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2023-07-10T17:16:27.810 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2023-07-10T17:16:27.810 INFO:teuthology.orchestra.run.smithi116.stdout: daemon iscsi.foo.smithi116.fkthyl on smithi116 is in error state 2023-07-10T17:16:27.811 INFO:teuthology.orchestra.run.smithi116.stdout: daemon prometheus.a on smithi192 is in error state 2023-07-10T17:16:28.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:28 smithi192 bash[50362]: audit 2023-07-10T17:16:26.539417+0000 mgr.y (mgr.25390) 379 : audit 0 from='client.44782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:16:28.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:28 smithi192 bash[50362]: audit 2023-07-10T17:16:27.230845+0000 mon.a (mon.0) 925 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:28.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:28 smithi192 bash[50362]: audit 2023-07-10T17:16:27.238273+0000 mon.a (mon.0) 926 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:28.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:28 smithi192 bash[50362]: audit 2023-07-10T17:16:27.671355+0000 mon.a (mon.0) 927 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/1249272063"}]': finished 2023-07-10T17:16:28.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:28 smithi192 bash[50362]: cluster 2023-07-10T17:16:27.671498+0000 mon.a (mon.0) 928 : cluster 0 osdmap e129: 8 total, 8 up, 8 in 2023-07-10T17:16:28.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:28 smithi192 bash[50362]: audit 2023-07-10T17:16:27.805727+0000 mon.a (mon.0) 929 : audit 0 from='client.44806 172.21.15.116:0/1754592127' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:16:28.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:28 smithi192 bash[50362]: audit 2023-07-10T17:16:28.027001+0000 mon.c (mon.1) 28 : audit 1 from='client.? 172.21.15.116:0/3025992572' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/48925241"}]: dispatch 2023-07-10T17:16:28.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:28 smithi192 bash[50362]: audit 2023-07-10T17:16:28.027858+0000 mon.a (mon.0) 930 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/48925241"}]: dispatch 2023-07-10T17:16:28.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:28 smithi116 bash[63900]: audit 2023-07-10T17:16:26.539417+0000 mgr.y (mgr.25390) 379 : audit 0 from='client.44782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:16:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:28 smithi116 bash[63900]: audit 2023-07-10T17:16:27.230845+0000 mon.a (mon.0) 925 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:28 smithi116 bash[63900]: audit 2023-07-10T17:16:27.238273+0000 mon.a (mon.0) 926 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:28 smithi116 bash[63900]: audit 2023-07-10T17:16:27.671355+0000 mon.a (mon.0) 927 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/1249272063"}]': finished 2023-07-10T17:16:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:28 smithi116 bash[63900]: cluster 2023-07-10T17:16:27.671498+0000 mon.a (mon.0) 928 : cluster 0 osdmap e129: 8 total, 8 up, 8 in 2023-07-10T17:16:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:28 smithi116 bash[63900]: audit 2023-07-10T17:16:27.805727+0000 mon.a (mon.0) 929 : audit 0 from='client.44806 172.21.15.116:0/1754592127' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:16:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:28 smithi116 bash[63900]: audit 2023-07-10T17:16:28.027001+0000 mon.c (mon.1) 28 : audit 1 from='client.? 172.21.15.116:0/3025992572' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/48925241"}]: dispatch 2023-07-10T17:16:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:28 smithi116 bash[63900]: audit 2023-07-10T17:16:28.027858+0000 mon.a (mon.0) 930 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/48925241"}]: dispatch 2023-07-10T17:16:28.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:28 smithi116 bash[65980]: audit 2023-07-10T17:16:26.539417+0000 mgr.y (mgr.25390) 379 : audit 0 from='client.44782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:16:28.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:28 smithi116 bash[65980]: audit 2023-07-10T17:16:27.230845+0000 mon.a (mon.0) 925 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:28.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:28 smithi116 bash[65980]: audit 2023-07-10T17:16:27.238273+0000 mon.a (mon.0) 926 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:28.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:28 smithi116 bash[65980]: audit 2023-07-10T17:16:27.671355+0000 mon.a (mon.0) 927 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/1249272063"}]': finished 2023-07-10T17:16:28.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:28 smithi116 bash[65980]: cluster 2023-07-10T17:16:27.671498+0000 mon.a (mon.0) 928 : cluster 0 osdmap e129: 8 total, 8 up, 8 in 2023-07-10T17:16:28.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:28 smithi116 bash[65980]: audit 2023-07-10T17:16:27.805727+0000 mon.a (mon.0) 929 : audit 0 from='client.44806 172.21.15.116:0/1754592127' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:16:28.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:28 smithi116 bash[65980]: audit 2023-07-10T17:16:28.027001+0000 mon.c (mon.1) 28 : audit 1 from='client.? 172.21.15.116:0/3025992572' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/48925241"}]: dispatch 2023-07-10T17:16:28.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:28 smithi116 bash[65980]: audit 2023-07-10T17:16:28.027858+0000 mon.a (mon.0) 930 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/48925241"}]: dispatch 2023-07-10T17:16:29.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:29 smithi192 bash[50362]: audit 2023-07-10T17:16:27.363971+0000 mgr.y (mgr.25390) 380 : audit 0 from='client.34941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:16:29.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:29 smithi192 bash[50362]: cluster 2023-07-10T17:16:27.734814+0000 mgr.y (mgr.25390) 381 : cluster 0 pgmap v275: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:16:29.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:29 smithi192 bash[50362]: audit 2023-07-10T17:16:28.678596+0000 mon.a (mon.0) 931 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/48925241"}]': finished 2023-07-10T17:16:29.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:29 smithi192 bash[50362]: cluster 2023-07-10T17:16:28.678694+0000 mon.a (mon.0) 932 : cluster 0 osdmap e130: 8 total, 8 up, 8 in 2023-07-10T17:16:29.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:29 smithi192 bash[50362]: audit 2023-07-10T17:16:29.067702+0000 mon.a (mon.0) 933 : audit 1 from='client.? 172.21.15.116:0/2471246024' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/670271520"}]: dispatch 2023-07-10T17:16:29.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:29 smithi116 bash[65980]: audit 2023-07-10T17:16:27.363971+0000 mgr.y (mgr.25390) 380 : audit 0 from='client.34941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:16:29.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:29 smithi116 bash[65980]: cluster 2023-07-10T17:16:27.734814+0000 mgr.y (mgr.25390) 381 : cluster 0 pgmap v275: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:16:29.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:29 smithi116 bash[65980]: audit 2023-07-10T17:16:28.678596+0000 mon.a (mon.0) 931 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/48925241"}]': finished 2023-07-10T17:16:29.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:29 smithi116 bash[65980]: cluster 2023-07-10T17:16:28.678694+0000 mon.a (mon.0) 932 : cluster 0 osdmap e130: 8 total, 8 up, 8 in 2023-07-10T17:16:29.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:29 smithi116 bash[65980]: audit 2023-07-10T17:16:29.067702+0000 mon.a (mon.0) 933 : audit 1 from='client.? 172.21.15.116:0/2471246024' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/670271520"}]: dispatch 2023-07-10T17:16:29.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:29 smithi116 bash[63900]: audit 2023-07-10T17:16:27.363971+0000 mgr.y (mgr.25390) 380 : audit 0 from='client.34941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:16:29.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:29 smithi116 bash[63900]: cluster 2023-07-10T17:16:27.734814+0000 mgr.y (mgr.25390) 381 : cluster 0 pgmap v275: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:16:29.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:29 smithi116 bash[63900]: audit 2023-07-10T17:16:28.678596+0000 mon.a (mon.0) 931 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/48925241"}]': finished 2023-07-10T17:16:29.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:29 smithi116 bash[63900]: cluster 2023-07-10T17:16:28.678694+0000 mon.a (mon.0) 932 : cluster 0 osdmap e130: 8 total, 8 up, 8 in 2023-07-10T17:16:29.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:29 smithi116 bash[63900]: audit 2023-07-10T17:16:29.067702+0000 mon.a (mon.0) 933 : audit 1 from='client.? 172.21.15.116:0/2471246024' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/670271520"}]: dispatch 2023-07-10T17:16:31.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:30 smithi192 bash[50362]: audit 2023-07-10T17:16:29.685821+0000 mon.a (mon.0) 934 : audit 1 from='client.? 172.21.15.116:0/2471246024' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/670271520"}]': finished 2023-07-10T17:16:31.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:30 smithi192 bash[50362]: cluster 2023-07-10T17:16:29.685994+0000 mon.a (mon.0) 935 : cluster 0 osdmap e131: 8 total, 8 up, 8 in 2023-07-10T17:16:31.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:30 smithi192 bash[50362]: cluster 2023-07-10T17:16:29.735503+0000 mgr.y (mgr.25390) 382 : cluster 0 pgmap v278: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:16:31.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:30 smithi192 bash[50362]: audit 2023-07-10T17:16:30.073753+0000 mon.c (mon.1) 29 : audit 1 from='client.? 172.21.15.116:0/1414862633' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1075601410"}]: dispatch 2023-07-10T17:16:31.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:30 smithi192 bash[50362]: audit 2023-07-10T17:16:30.074671+0000 mon.a (mon.0) 936 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1075601410"}]: dispatch 2023-07-10T17:16:31.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:30 smithi116 bash[63900]: audit 2023-07-10T17:16:29.685821+0000 mon.a (mon.0) 934 : audit 1 from='client.? 172.21.15.116:0/2471246024' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/670271520"}]': finished 2023-07-10T17:16:31.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:30 smithi116 bash[63900]: cluster 2023-07-10T17:16:29.685994+0000 mon.a (mon.0) 935 : cluster 0 osdmap e131: 8 total, 8 up, 8 in 2023-07-10T17:16:31.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:30 smithi116 bash[63900]: cluster 2023-07-10T17:16:29.735503+0000 mgr.y (mgr.25390) 382 : cluster 0 pgmap v278: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:16:31.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:30 smithi116 bash[63900]: audit 2023-07-10T17:16:30.073753+0000 mon.c (mon.1) 29 : audit 1 from='client.? 172.21.15.116:0/1414862633' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1075601410"}]: dispatch 2023-07-10T17:16:31.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:30 smithi116 bash[63900]: audit 2023-07-10T17:16:30.074671+0000 mon.a (mon.0) 936 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1075601410"}]: dispatch 2023-07-10T17:16:31.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:30 smithi116 bash[65980]: audit 2023-07-10T17:16:29.685821+0000 mon.a (mon.0) 934 : audit 1 from='client.? 172.21.15.116:0/2471246024' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/670271520"}]': finished 2023-07-10T17:16:31.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:30 smithi116 bash[65980]: cluster 2023-07-10T17:16:29.685994+0000 mon.a (mon.0) 935 : cluster 0 osdmap e131: 8 total, 8 up, 8 in 2023-07-10T17:16:31.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:30 smithi116 bash[65980]: cluster 2023-07-10T17:16:29.735503+0000 mgr.y (mgr.25390) 382 : cluster 0 pgmap v278: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:16:31.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:30 smithi116 bash[65980]: audit 2023-07-10T17:16:30.073753+0000 mon.c (mon.1) 29 : audit 1 from='client.? 172.21.15.116:0/1414862633' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1075601410"}]: dispatch 2023-07-10T17:16:31.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:30 smithi116 bash[65980]: audit 2023-07-10T17:16:30.074671+0000 mon.a (mon.0) 936 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1075601410"}]: dispatch 2023-07-10T17:16:32.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:31 smithi192 bash[50362]: audit 2023-07-10T17:16:30.692855+0000 mon.a (mon.0) 937 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1075601410"}]': finished 2023-07-10T17:16:32.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:31 smithi192 bash[50362]: cluster 2023-07-10T17:16:30.692960+0000 mon.a (mon.0) 938 : cluster 0 osdmap e132: 8 total, 8 up, 8 in 2023-07-10T17:16:32.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:31 smithi192 bash[50362]: audit 2023-07-10T17:16:31.082091+0000 mon.c (mon.1) 30 : audit 1 from='client.? 172.21.15.116:0/2330533981' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: dispatch 2023-07-10T17:16:32.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:31 smithi192 bash[50362]: audit 2023-07-10T17:16:31.083016+0000 mon.a (mon.0) 939 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: dispatch 2023-07-10T17:16:32.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:31 smithi192 bash[50362]: audit 2023-07-10T17:16:31.169806+0000 mgr.y (mgr.25390) 383 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:16:32.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:31 smithi116 bash[63900]: audit 2023-07-10T17:16:30.692855+0000 mon.a (mon.0) 937 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1075601410"}]': finished 2023-07-10T17:16:32.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:31 smithi116 bash[63900]: cluster 2023-07-10T17:16:30.692960+0000 mon.a (mon.0) 938 : cluster 0 osdmap e132: 8 total, 8 up, 8 in 2023-07-10T17:16:32.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:31 smithi116 bash[63900]: audit 2023-07-10T17:16:31.082091+0000 mon.c (mon.1) 30 : audit 1 from='client.? 172.21.15.116:0/2330533981' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: dispatch 2023-07-10T17:16:32.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:31 smithi116 bash[63900]: audit 2023-07-10T17:16:31.083016+0000 mon.a (mon.0) 939 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: dispatch 2023-07-10T17:16:32.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:31 smithi116 bash[63900]: audit 2023-07-10T17:16:31.169806+0000 mgr.y (mgr.25390) 383 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:16:32.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:31 smithi116 bash[65980]: audit 2023-07-10T17:16:30.692855+0000 mon.a (mon.0) 937 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1075601410"}]': finished 2023-07-10T17:16:32.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:31 smithi116 bash[65980]: cluster 2023-07-10T17:16:30.692960+0000 mon.a (mon.0) 938 : cluster 0 osdmap e132: 8 total, 8 up, 8 in 2023-07-10T17:16:32.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:31 smithi116 bash[65980]: audit 2023-07-10T17:16:31.082091+0000 mon.c (mon.1) 30 : audit 1 from='client.? 172.21.15.116:0/2330533981' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: dispatch 2023-07-10T17:16:32.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:31 smithi116 bash[65980]: audit 2023-07-10T17:16:31.083016+0000 mon.a (mon.0) 939 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]: dispatch 2023-07-10T17:16:32.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:31 smithi116 bash[65980]: audit 2023-07-10T17:16:31.169806+0000 mgr.y (mgr.25390) 383 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:16:33.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:32 smithi192 bash[50362]: audit 2023-07-10T17:16:31.701698+0000 mon.a (mon.0) 940 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]': finished 2023-07-10T17:16:33.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:32 smithi192 bash[50362]: cluster 2023-07-10T17:16:31.701796+0000 mon.a (mon.0) 941 : cluster 0 osdmap e133: 8 total, 8 up, 8 in 2023-07-10T17:16:33.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:32 smithi192 bash[50362]: cluster 2023-07-10T17:16:31.736127+0000 mgr.y (mgr.25390) 384 : cluster 0 pgmap v281: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:16:33.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:32 smithi192 bash[50362]: audit 2023-07-10T17:16:32.092650+0000 mon.a (mon.0) 942 : audit 1 from='client.? 172.21.15.116:0/4069056161' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/234488842"}]: dispatch 2023-07-10T17:16:33.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:32 smithi116 bash[65980]: audit 2023-07-10T17:16:31.701698+0000 mon.a (mon.0) 940 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]': finished 2023-07-10T17:16:33.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:32 smithi116 bash[65980]: cluster 2023-07-10T17:16:31.701796+0000 mon.a (mon.0) 941 : cluster 0 osdmap e133: 8 total, 8 up, 8 in 2023-07-10T17:16:33.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:32 smithi116 bash[65980]: cluster 2023-07-10T17:16:31.736127+0000 mgr.y (mgr.25390) 384 : cluster 0 pgmap v281: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:16:33.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:32 smithi116 bash[65980]: audit 2023-07-10T17:16:32.092650+0000 mon.a (mon.0) 942 : audit 1 from='client.? 172.21.15.116:0/4069056161' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/234488842"}]: dispatch 2023-07-10T17:16:33.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:32 smithi116 bash[63900]: audit 2023-07-10T17:16:31.701698+0000 mon.a (mon.0) 940 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/670271520"}]': finished 2023-07-10T17:16:33.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:32 smithi116 bash[63900]: cluster 2023-07-10T17:16:31.701796+0000 mon.a (mon.0) 941 : cluster 0 osdmap e133: 8 total, 8 up, 8 in 2023-07-10T17:16:33.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:32 smithi116 bash[63900]: cluster 2023-07-10T17:16:31.736127+0000 mgr.y (mgr.25390) 384 : cluster 0 pgmap v281: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:16:33.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:32 smithi116 bash[63900]: audit 2023-07-10T17:16:32.092650+0000 mon.a (mon.0) 942 : audit 1 from='client.? 172.21.15.116:0/4069056161' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/234488842"}]: dispatch 2023-07-10T17:16:33.834 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:33 smithi116 bash[63900]: audit 2023-07-10T17:16:32.707392+0000 mon.a (mon.0) 943 : audit 1 from='client.? 172.21.15.116:0/4069056161' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/234488842"}]': finished 2023-07-10T17:16:33.835 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:33 smithi116 bash[63900]: cluster 2023-07-10T17:16:32.707588+0000 mon.a (mon.0) 944 : cluster 0 osdmap e134: 8 total, 8 up, 8 in 2023-07-10T17:16:33.835 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:33 smithi116 bash[63900]: audit 2023-07-10T17:16:33.093687+0000 mon.a (mon.0) 945 : audit 1 from='client.? 172.21.15.116:0/1808058164' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/48925241"}]: dispatch 2023-07-10T17:16:33.835 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:33 smithi116 bash[65980]: audit 2023-07-10T17:16:32.707392+0000 mon.a (mon.0) 943 : audit 1 from='client.? 172.21.15.116:0/4069056161' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/234488842"}]': finished 2023-07-10T17:16:33.835 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:33 smithi116 bash[65980]: cluster 2023-07-10T17:16:32.707588+0000 mon.a (mon.0) 944 : cluster 0 osdmap e134: 8 total, 8 up, 8 in 2023-07-10T17:16:33.835 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:33 smithi116 bash[65980]: audit 2023-07-10T17:16:33.093687+0000 mon.a (mon.0) 945 : audit 1 from='client.? 172.21.15.116:0/1808058164' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/48925241"}]: dispatch 2023-07-10T17:16:34.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:33 smithi192 bash[50362]: audit 2023-07-10T17:16:32.707392+0000 mon.a (mon.0) 943 : audit 1 from='client.? 172.21.15.116:0/4069056161' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/234488842"}]': finished 2023-07-10T17:16:34.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:33 smithi192 bash[50362]: cluster 2023-07-10T17:16:32.707588+0000 mon.a (mon.0) 944 : cluster 0 osdmap e134: 8 total, 8 up, 8 in 2023-07-10T17:16:34.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:33 smithi192 bash[50362]: audit 2023-07-10T17:16:33.093687+0000 mon.a (mon.0) 945 : audit 1 from='client.? 172.21.15.116:0/1808058164' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/48925241"}]: dispatch 2023-07-10T17:16:34.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:34 smithi116 bash[65980]: audit 2023-07-10T17:16:33.718152+0000 mon.a (mon.0) 946 : audit 1 from='client.? 172.21.15.116:0/1808058164' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/48925241"}]': finished 2023-07-10T17:16:34.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:34 smithi116 bash[65980]: cluster 2023-07-10T17:16:33.718247+0000 mon.a (mon.0) 947 : cluster 0 osdmap e135: 8 total, 8 up, 8 in 2023-07-10T17:16:34.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:34 smithi116 bash[65980]: cluster 2023-07-10T17:16:33.736576+0000 mgr.y (mgr.25390) 385 : cluster 0 pgmap v284: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:16:34.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:34 smithi116 bash[65980]: audit 2023-07-10T17:16:34.116846+0000 mon.a (mon.0) 948 : audit 1 from='client.? 172.21.15.116:0/3637040822' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1382514600"}]: dispatch 2023-07-10T17:16:34.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:34 smithi116 bash[63900]: audit 2023-07-10T17:16:33.718152+0000 mon.a (mon.0) 946 : audit 1 from='client.? 172.21.15.116:0/1808058164' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/48925241"}]': finished 2023-07-10T17:16:34.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:34 smithi116 bash[63900]: cluster 2023-07-10T17:16:33.718247+0000 mon.a (mon.0) 947 : cluster 0 osdmap e135: 8 total, 8 up, 8 in 2023-07-10T17:16:34.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:34 smithi116 bash[63900]: cluster 2023-07-10T17:16:33.736576+0000 mgr.y (mgr.25390) 385 : cluster 0 pgmap v284: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:16:34.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:34 smithi116 bash[63900]: audit 2023-07-10T17:16:34.116846+0000 mon.a (mon.0) 948 : audit 1 from='client.? 172.21.15.116:0/3637040822' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1382514600"}]: dispatch 2023-07-10T17:16:35.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:34 smithi192 bash[50362]: audit 2023-07-10T17:16:33.718152+0000 mon.a (mon.0) 946 : audit 1 from='client.? 172.21.15.116:0/1808058164' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/48925241"}]': finished 2023-07-10T17:16:35.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:34 smithi192 bash[50362]: cluster 2023-07-10T17:16:33.718247+0000 mon.a (mon.0) 947 : cluster 0 osdmap e135: 8 total, 8 up, 8 in 2023-07-10T17:16:35.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:34 smithi192 bash[50362]: cluster 2023-07-10T17:16:33.736576+0000 mgr.y (mgr.25390) 385 : cluster 0 pgmap v284: 161 pgs: 161 active+clean; 604 KiB data, 325 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:16:35.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:34 smithi192 bash[50362]: audit 2023-07-10T17:16:34.116846+0000 mon.a (mon.0) 948 : audit 1 from='client.? 172.21.15.116:0/3637040822' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1382514600"}]: dispatch 2023-07-10T17:16:35.771 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:35 smithi116 bash[63900]: audit 2023-07-10T17:16:34.739201+0000 mon.a (mon.0) 949 : audit 1 from='client.? 172.21.15.116:0/3637040822' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1382514600"}]': finished 2023-07-10T17:16:35.772 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:35 smithi116 bash[63900]: cluster 2023-07-10T17:16:34.739275+0000 mon.a (mon.0) 950 : cluster 0 osdmap e136: 8 total, 8 up, 8 in 2023-07-10T17:16:35.772 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:35 smithi116 bash[63900]: audit 2023-07-10T17:16:35.126257+0000 mon.a (mon.0) 951 : audit 1 from='client.? 172.21.15.116:0/3907231615' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3314203798"}]: dispatch 2023-07-10T17:16:35.772 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:35 smithi116 bash[65980]: audit 2023-07-10T17:16:34.739201+0000 mon.a (mon.0) 949 : audit 1 from='client.? 172.21.15.116:0/3637040822' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1382514600"}]': finished 2023-07-10T17:16:35.772 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:35 smithi116 bash[65980]: cluster 2023-07-10T17:16:34.739275+0000 mon.a (mon.0) 950 : cluster 0 osdmap e136: 8 total, 8 up, 8 in 2023-07-10T17:16:35.773 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:35 smithi116 bash[65980]: audit 2023-07-10T17:16:35.126257+0000 mon.a (mon.0) 951 : audit 1 from='client.? 172.21.15.116:0/3907231615' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3314203798"}]: dispatch 2023-07-10T17:16:36.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:35 smithi192 bash[50362]: audit 2023-07-10T17:16:34.739201+0000 mon.a (mon.0) 949 : audit 1 from='client.? 172.21.15.116:0/3637040822' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/1382514600"}]': finished 2023-07-10T17:16:36.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:35 smithi192 bash[50362]: cluster 2023-07-10T17:16:34.739275+0000 mon.a (mon.0) 950 : cluster 0 osdmap e136: 8 total, 8 up, 8 in 2023-07-10T17:16:36.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:35 smithi192 bash[50362]: audit 2023-07-10T17:16:35.126257+0000 mon.a (mon.0) 951 : audit 1 from='client.? 172.21.15.116:0/3907231615' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3314203798"}]: dispatch 2023-07-10T17:16:36.784 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:36 smithi116 bash[63900]: cluster 2023-07-10T17:16:35.737876+0000 mgr.y (mgr.25390) 386 : cluster 0 pgmap v286: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:36.784 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:36 smithi116 bash[63900]: audit 2023-07-10T17:16:35.750490+0000 mon.a (mon.0) 952 : audit 1 from='client.? 172.21.15.116:0/3907231615' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3314203798"}]': finished 2023-07-10T17:16:36.784 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:36 smithi116 bash[63900]: cluster 2023-07-10T17:16:35.750711+0000 mon.a (mon.0) 953 : cluster 0 osdmap e137: 8 total, 8 up, 8 in 2023-07-10T17:16:36.784 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:36 smithi116 bash[63900]: audit 2023-07-10T17:16:36.133245+0000 mon.c (mon.1) 31 : audit 1 from='client.? 172.21.15.116:0/126215509' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/3262996548"}]: dispatch 2023-07-10T17:16:36.785 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:36 smithi116 bash[63900]: audit 2023-07-10T17:16:36.134122+0000 mon.a (mon.0) 954 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/3262996548"}]: dispatch 2023-07-10T17:16:36.785 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:36 smithi116 bash[65980]: cluster 2023-07-10T17:16:35.737876+0000 mgr.y (mgr.25390) 386 : cluster 0 pgmap v286: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:36.785 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:36 smithi116 bash[65980]: audit 2023-07-10T17:16:35.750490+0000 mon.a (mon.0) 952 : audit 1 from='client.? 172.21.15.116:0/3907231615' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3314203798"}]': finished 2023-07-10T17:16:36.785 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:36 smithi116 bash[65980]: cluster 2023-07-10T17:16:35.750711+0000 mon.a (mon.0) 953 : cluster 0 osdmap e137: 8 total, 8 up, 8 in 2023-07-10T17:16:36.785 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:36 smithi116 bash[65980]: audit 2023-07-10T17:16:36.133245+0000 mon.c (mon.1) 31 : audit 1 from='client.? 172.21.15.116:0/126215509' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/3262996548"}]: dispatch 2023-07-10T17:16:36.786 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:36 smithi116 bash[65980]: audit 2023-07-10T17:16:36.134122+0000 mon.a (mon.0) 954 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/3262996548"}]: dispatch 2023-07-10T17:16:37.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:36 smithi192 bash[50362]: cluster 2023-07-10T17:16:35.737876+0000 mgr.y (mgr.25390) 386 : cluster 0 pgmap v286: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail 2023-07-10T17:16:37.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:36 smithi192 bash[50362]: audit 2023-07-10T17:16:35.750490+0000 mon.a (mon.0) 952 : audit 1 from='client.? 172.21.15.116:0/3907231615' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3314203798"}]': finished 2023-07-10T17:16:37.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:36 smithi192 bash[50362]: cluster 2023-07-10T17:16:35.750711+0000 mon.a (mon.0) 953 : cluster 0 osdmap e137: 8 total, 8 up, 8 in 2023-07-10T17:16:37.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:36 smithi192 bash[50362]: audit 2023-07-10T17:16:36.133245+0000 mon.c (mon.1) 31 : audit 1 from='client.? 172.21.15.116:0/126215509' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/3262996548"}]: dispatch 2023-07-10T17:16:37.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:36 smithi192 bash[50362]: audit 2023-07-10T17:16:36.134122+0000 mon.a (mon.0) 954 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/3262996548"}]: dispatch 2023-07-10T17:16:38.120 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: audit 2023-07-10T17:16:36.761360+0000 mon.a (mon.0) 955 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/3262996548"}]': finished 2023-07-10T17:16:38.120 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: cluster 2023-07-10T17:16:36.761530+0000 mon.a (mon.0) 956 : cluster 0 osdmap e138: 8 total, 8 up, 8 in 2023-07-10T17:16:38.121 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: audit 2023-07-10T17:16:37.087608+0000 mon.a (mon.0) 957 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:38.121 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: audit 2023-07-10T17:16:37.097663+0000 mon.a (mon.0) 958 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:38.121 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: audit 2023-07-10T17:16:37.099375+0000 mon.a (mon.0) 959 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:16:38.121 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: audit 2023-07-10T17:16:37.100450+0000 mon.a (mon.0) 960 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:16:38.121 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: cephadm 2023-07-10T17:16:37.101395+0000 mgr.y (mgr.25390) 387 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:16:38.122 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: cluster 2023-07-10T17:16:37.102564+0000 mgr.y (mgr.25390) 388 : cluster 0 pgmap v289: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-10T17:16:38.122 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: cluster 2023-07-10T17:16:37.103201+0000 mgr.y (mgr.25390) 389 : cluster 0 pgmap v290: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-10T17:16:38.122 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: cephadm 2023-07-10T17:16:37.107053+0000 mgr.y (mgr.25390) 390 : cephadm 1 Checking dashboard <-> RGW credentials 2023-07-10T17:16:38.122 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: audit 2023-07-10T17:16:37.159702+0000 mon.c (mon.1) 32 : audit 1 from='client.? 172.21.15.116:0/17496995' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/3262996548"}]: dispatch 2023-07-10T17:16:38.122 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: audit 2023-07-10T17:16:37.160334+0000 mon.a (mon.0) 961 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/3262996548"}]: dispatch 2023-07-10T17:16:38.122 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: audit 2023-07-10T17:16:37.578000+0000 mon.a (mon.0) 962 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:38.122 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: audit 2023-07-10T17:16:37.592340+0000 mon.a (mon.0) 963 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:38.123 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: audit 2023-07-10T17:16:37.601911+0000 mon.a (mon.0) 964 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:38.123 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: audit 2023-07-10T17:16:37.721807+0000 mon.a (mon.0) 965 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:16:38.123 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: audit 2023-07-10T17:16:37.731292+0000 mon.a (mon.0) 966 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:16:38.123 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: audit 2023-07-10T17:16:37.744048+0000 mon.a (mon.0) 967 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:16:38.123 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: audit 2023-07-10T17:16:37.749074+0000 mon.a (mon.0) 968 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:38.123 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: audit 2023-07-10T17:16:37.751572+0000 mon.a (mon.0) 969 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:38.123 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[65980]: audit 2023-07-10T17:16:37.753550+0000 mon.a (mon.0) 970 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:38.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: audit 2023-07-10T17:16:36.761360+0000 mon.a (mon.0) 955 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/3262996548"}]': finished 2023-07-10T17:16:38.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: cluster 2023-07-10T17:16:36.761530+0000 mon.a (mon.0) 956 : cluster 0 osdmap e138: 8 total, 8 up, 8 in 2023-07-10T17:16:38.125 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: audit 2023-07-10T17:16:37.087608+0000 mon.a (mon.0) 957 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:38.125 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: audit 2023-07-10T17:16:37.097663+0000 mon.a (mon.0) 958 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:38.125 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: audit 2023-07-10T17:16:37.099375+0000 mon.a (mon.0) 959 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:16:38.126 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: audit 2023-07-10T17:16:37.100450+0000 mon.a (mon.0) 960 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:16:38.126 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: cephadm 2023-07-10T17:16:37.101395+0000 mgr.y (mgr.25390) 387 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:16:38.126 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: cluster 2023-07-10T17:16:37.102564+0000 mgr.y (mgr.25390) 388 : cluster 0 pgmap v289: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-10T17:16:38.126 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: cluster 2023-07-10T17:16:37.103201+0000 mgr.y (mgr.25390) 389 : cluster 0 pgmap v290: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-10T17:16:38.127 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: cephadm 2023-07-10T17:16:37.107053+0000 mgr.y (mgr.25390) 390 : cephadm 1 Checking dashboard <-> RGW credentials 2023-07-10T17:16:38.127 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: audit 2023-07-10T17:16:37.159702+0000 mon.c (mon.1) 32 : audit 1 from='client.? 172.21.15.116:0/17496995' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/3262996548"}]: dispatch 2023-07-10T17:16:38.127 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: audit 2023-07-10T17:16:37.160334+0000 mon.a (mon.0) 961 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/3262996548"}]: dispatch 2023-07-10T17:16:38.128 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: audit 2023-07-10T17:16:37.578000+0000 mon.a (mon.0) 962 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:38.128 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: audit 2023-07-10T17:16:37.592340+0000 mon.a (mon.0) 963 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:38.129 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: audit 2023-07-10T17:16:37.601911+0000 mon.a (mon.0) 964 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:38.129 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: audit 2023-07-10T17:16:37.721807+0000 mon.a (mon.0) 965 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:16:38.129 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: audit 2023-07-10T17:16:37.731292+0000 mon.a (mon.0) 966 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:16:38.130 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: audit 2023-07-10T17:16:37.744048+0000 mon.a (mon.0) 967 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:16:38.130 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: audit 2023-07-10T17:16:37.749074+0000 mon.a (mon.0) 968 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:38.131 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: audit 2023-07-10T17:16:37.751572+0000 mon.a (mon.0) 969 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:38.131 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:37 smithi116 bash[63900]: audit 2023-07-10T17:16:37.753550+0000 mon.a (mon.0) 970 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:38.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: audit 2023-07-10T17:16:36.761360+0000 mon.a (mon.0) 955 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6800/3262996548"}]': finished 2023-07-10T17:16:38.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: cluster 2023-07-10T17:16:36.761530+0000 mon.a (mon.0) 956 : cluster 0 osdmap e138: 8 total, 8 up, 8 in 2023-07-10T17:16:38.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: audit 2023-07-10T17:16:37.087608+0000 mon.a (mon.0) 957 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:38.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: audit 2023-07-10T17:16:37.097663+0000 mon.a (mon.0) 958 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:38.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: audit 2023-07-10T17:16:37.099375+0000 mon.a (mon.0) 959 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:16:38.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: audit 2023-07-10T17:16:37.100450+0000 mon.a (mon.0) 960 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:16:38.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: cephadm 2023-07-10T17:16:37.101395+0000 mgr.y (mgr.25390) 387 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:16:38.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: cluster 2023-07-10T17:16:37.102564+0000 mgr.y (mgr.25390) 388 : cluster 0 pgmap v289: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-10T17:16:38.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: cluster 2023-07-10T17:16:37.103201+0000 mgr.y (mgr.25390) 389 : cluster 0 pgmap v290: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-10T17:16:38.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: cephadm 2023-07-10T17:16:37.107053+0000 mgr.y (mgr.25390) 390 : cephadm 1 Checking dashboard <-> RGW credentials 2023-07-10T17:16:38.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: audit 2023-07-10T17:16:37.159702+0000 mon.c (mon.1) 32 : audit 1 from='client.? 172.21.15.116:0/17496995' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/3262996548"}]: dispatch 2023-07-10T17:16:38.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: audit 2023-07-10T17:16:37.160334+0000 mon.a (mon.0) 961 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/3262996548"}]: dispatch 2023-07-10T17:16:38.141 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: audit 2023-07-10T17:16:37.578000+0000 mon.a (mon.0) 962 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:38.141 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: audit 2023-07-10T17:16:37.592340+0000 mon.a (mon.0) 963 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:38.141 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: audit 2023-07-10T17:16:37.601911+0000 mon.a (mon.0) 964 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:38.141 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: audit 2023-07-10T17:16:37.721807+0000 mon.a (mon.0) 965 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:16:38.142 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: audit 2023-07-10T17:16:37.731292+0000 mon.a (mon.0) 966 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:16:38.142 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: audit 2023-07-10T17:16:37.744048+0000 mon.a (mon.0) 967 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:16:38.142 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: audit 2023-07-10T17:16:37.749074+0000 mon.a (mon.0) 968 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:38.142 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: audit 2023-07-10T17:16:37.751572+0000 mon.a (mon.0) 969 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:38.143 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:37 smithi192 bash[50362]: audit 2023-07-10T17:16:37.753550+0000 mon.a (mon.0) 970 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.122 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: audit 2023-07-10T17:16:37.722696+0000 mgr.y (mgr.25390) 391 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:16:39.122 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: audit 2023-07-10T17:16:37.731938+0000 mgr.y (mgr.25390) 392 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:16:39.123 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: audit 2023-07-10T17:16:37.763510+0000 mon.a (mon.0) 971 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.123 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: audit 2023-07-10T17:16:37.768169+0000 mon.a (mon.0) 972 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.123 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: audit 2023-07-10T17:16:37.770490+0000 mon.a (mon.0) 973 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.123 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: audit 2023-07-10T17:16:37.772027+0000 mon.a (mon.0) 974 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.123 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: audit 2023-07-10T17:16:37.773583+0000 mon.a (mon.0) 975 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.123 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: audit 2023-07-10T17:16:37.775107+0000 mon.a (mon.0) 976 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: audit 2023-07-10T17:16:37.777011+0000 mon.a (mon.0) 977 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: cephadm 2023-07-10T17:16:37.777979+0000 mgr.y (mgr.25390) 393 : cephadm 1 Upgrade: Setting container_image for all iscsi 2023-07-10T17:16:39.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: audit 2023-07-10T17:16:37.787128+0000 mon.a (mon.0) 978 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:39.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: audit 2023-07-10T17:16:37.788528+0000 mon.a (mon.0) 979 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi116.fkthyl"}]: dispatch 2023-07-10T17:16:39.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: audit 2023-07-10T17:16:37.797097+0000 mon.a (mon.0) 980 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi116.fkthyl"}]': finished 2023-07-10T17:16:39.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: audit 2023-07-10T17:16:37.799819+0000 mon.a (mon.0) 981 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: cephadm 2023-07-10T17:16:37.801075+0000 mgr.y (mgr.25390) 394 : cephadm 1 Upgrade: Setting container_image for all nfs 2023-07-10T17:16:39.125 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: audit 2023-07-10T17:16:37.811326+0000 mon.a (mon.0) 982 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:39.125 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: cluster 2023-07-10T17:16:38.094722+0000 mon.a (mon.0) 983 : cluster 3 Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:16:39.125 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: audit 2023-07-10T17:16:38.112813+0000 mon.a (mon.0) 984 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/3262996548"}]': finished 2023-07-10T17:16:39.125 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: cluster 2023-07-10T17:16:38.112955+0000 mon.a (mon.0) 985 : cluster 0 osdmap e139: 8 total, 8 up, 8 in 2023-07-10T17:16:39.125 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[63900]: audit 2023-07-10T17:16:38.489827+0000 mon.a (mon.0) 986 : audit 1 from='client.? 172.21.15.116:0/4203715672' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/221490838"}]: dispatch 2023-07-10T17:16:39.125 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: audit 2023-07-10T17:16:37.722696+0000 mgr.y (mgr.25390) 391 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:16:39.126 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: audit 2023-07-10T17:16:37.731938+0000 mgr.y (mgr.25390) 392 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:16:39.126 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: audit 2023-07-10T17:16:37.763510+0000 mon.a (mon.0) 971 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.126 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: audit 2023-07-10T17:16:37.768169+0000 mon.a (mon.0) 972 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.126 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: audit 2023-07-10T17:16:37.770490+0000 mon.a (mon.0) 973 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.126 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: audit 2023-07-10T17:16:37.772027+0000 mon.a (mon.0) 974 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.126 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: audit 2023-07-10T17:16:37.773583+0000 mon.a (mon.0) 975 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.126 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: audit 2023-07-10T17:16:37.775107+0000 mon.a (mon.0) 976 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.126 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: audit 2023-07-10T17:16:37.777011+0000 mon.a (mon.0) 977 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.127 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: cephadm 2023-07-10T17:16:37.777979+0000 mgr.y (mgr.25390) 393 : cephadm 1 Upgrade: Setting container_image for all iscsi 2023-07-10T17:16:39.127 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: audit 2023-07-10T17:16:37.787128+0000 mon.a (mon.0) 978 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:39.127 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: audit 2023-07-10T17:16:37.788528+0000 mon.a (mon.0) 979 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi116.fkthyl"}]: dispatch 2023-07-10T17:16:39.127 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: audit 2023-07-10T17:16:37.797097+0000 mon.a (mon.0) 980 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi116.fkthyl"}]': finished 2023-07-10T17:16:39.127 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: audit 2023-07-10T17:16:37.799819+0000 mon.a (mon.0) 981 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.127 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: cephadm 2023-07-10T17:16:37.801075+0000 mgr.y (mgr.25390) 394 : cephadm 1 Upgrade: Setting container_image for all nfs 2023-07-10T17:16:39.127 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: audit 2023-07-10T17:16:37.811326+0000 mon.a (mon.0) 982 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:39.128 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: cluster 2023-07-10T17:16:38.094722+0000 mon.a (mon.0) 983 : cluster 3 Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:16:39.128 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: audit 2023-07-10T17:16:38.112813+0000 mon.a (mon.0) 984 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/3262996548"}]': finished 2023-07-10T17:16:39.128 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: cluster 2023-07-10T17:16:38.112955+0000 mon.a (mon.0) 985 : cluster 0 osdmap e139: 8 total, 8 up, 8 in 2023-07-10T17:16:39.128 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:38 smithi116 bash[65980]: audit 2023-07-10T17:16:38.489827+0000 mon.a (mon.0) 986 : audit 1 from='client.? 172.21.15.116:0/4203715672' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/221490838"}]: dispatch 2023-07-10T17:16:39.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: audit 2023-07-10T17:16:37.722696+0000 mgr.y (mgr.25390) 391 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-07-10T17:16:39.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: audit 2023-07-10T17:16:37.731938+0000 mgr.y (mgr.25390) 392 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-07-10T17:16:39.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: audit 2023-07-10T17:16:37.763510+0000 mon.a (mon.0) 971 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: audit 2023-07-10T17:16:37.768169+0000 mon.a (mon.0) 972 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: audit 2023-07-10T17:16:37.770490+0000 mon.a (mon.0) 973 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: audit 2023-07-10T17:16:37.772027+0000 mon.a (mon.0) 974 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: audit 2023-07-10T17:16:37.773583+0000 mon.a (mon.0) 975 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: audit 2023-07-10T17:16:37.775107+0000 mon.a (mon.0) 976 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: audit 2023-07-10T17:16:37.777011+0000 mon.a (mon.0) 977 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: cephadm 2023-07-10T17:16:37.777979+0000 mgr.y (mgr.25390) 393 : cephadm 1 Upgrade: Setting container_image for all iscsi 2023-07-10T17:16:39.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: audit 2023-07-10T17:16:37.787128+0000 mon.a (mon.0) 978 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:39.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: audit 2023-07-10T17:16:37.788528+0000 mon.a (mon.0) 979 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi116.fkthyl"}]: dispatch 2023-07-10T17:16:39.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: audit 2023-07-10T17:16:37.797097+0000 mon.a (mon.0) 980 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi116.fkthyl"}]': finished 2023-07-10T17:16:39.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: audit 2023-07-10T17:16:37.799819+0000 mon.a (mon.0) 981 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:16:39.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: cephadm 2023-07-10T17:16:37.801075+0000 mgr.y (mgr.25390) 394 : cephadm 1 Upgrade: Setting container_image for all nfs 2023-07-10T17:16:39.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: audit 2023-07-10T17:16:37.811326+0000 mon.a (mon.0) 982 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:39.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: cluster 2023-07-10T17:16:38.094722+0000 mon.a (mon.0) 983 : cluster 3 Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-10T17:16:39.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: audit 2023-07-10T17:16:38.112813+0000 mon.a (mon.0) 984 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:6801/3262996548"}]': finished 2023-07-10T17:16:39.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: cluster 2023-07-10T17:16:38.112955+0000 mon.a (mon.0) 985 : cluster 0 osdmap e139: 8 total, 8 up, 8 in 2023-07-10T17:16:39.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:38 smithi192 bash[50362]: audit 2023-07-10T17:16:38.489827+0000 mon.a (mon.0) 986 : audit 1 from='client.? 172.21.15.116:0/4203715672' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/221490838"}]: dispatch 2023-07-10T17:16:40.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:40 smithi192 bash[50362]: cluster 2023-07-10T17:16:39.107233+0000 mgr.y (mgr.25390) 395 : cluster 0 pgmap v292: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2023-07-10T17:16:40.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:40 smithi192 bash[50362]: audit 2023-07-10T17:16:39.117674+0000 mon.a (mon.0) 987 : audit 1 from='client.? 172.21.15.116:0/4203715672' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/221490838"}]': finished 2023-07-10T17:16:40.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:40 smithi192 bash[50362]: cluster 2023-07-10T17:16:39.117759+0000 mon.a (mon.0) 988 : cluster 0 osdmap e140: 8 total, 8 up, 8 in 2023-07-10T17:16:40.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:40 smithi192 bash[50362]: audit 2023-07-10T17:16:39.515850+0000 mon.a (mon.0) 989 : audit 1 from='client.? 172.21.15.116:0/1148106' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3104229546"}]: dispatch 2023-07-10T17:16:40.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:40 smithi116 bash[65980]: cluster 2023-07-10T17:16:39.107233+0000 mgr.y (mgr.25390) 395 : cluster 0 pgmap v292: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2023-07-10T17:16:40.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:40 smithi116 bash[65980]: audit 2023-07-10T17:16:39.117674+0000 mon.a (mon.0) 987 : audit 1 from='client.? 172.21.15.116:0/4203715672' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/221490838"}]': finished 2023-07-10T17:16:40.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:40 smithi116 bash[65980]: cluster 2023-07-10T17:16:39.117759+0000 mon.a (mon.0) 988 : cluster 0 osdmap e140: 8 total, 8 up, 8 in 2023-07-10T17:16:40.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:40 smithi116 bash[65980]: audit 2023-07-10T17:16:39.515850+0000 mon.a (mon.0) 989 : audit 1 from='client.? 172.21.15.116:0/1148106' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3104229546"}]: dispatch 2023-07-10T17:16:40.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:40 smithi116 bash[63900]: cluster 2023-07-10T17:16:39.107233+0000 mgr.y (mgr.25390) 395 : cluster 0 pgmap v292: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2023-07-10T17:16:40.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:40 smithi116 bash[63900]: audit 2023-07-10T17:16:39.117674+0000 mon.a (mon.0) 987 : audit 1 from='client.? 172.21.15.116:0/4203715672' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/221490838"}]': finished 2023-07-10T17:16:40.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:40 smithi116 bash[63900]: cluster 2023-07-10T17:16:39.117759+0000 mon.a (mon.0) 988 : cluster 0 osdmap e140: 8 total, 8 up, 8 in 2023-07-10T17:16:40.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:40 smithi116 bash[63900]: audit 2023-07-10T17:16:39.515850+0000 mon.a (mon.0) 989 : audit 1 from='client.? 172.21.15.116:0/1148106' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3104229546"}]: dispatch 2023-07-10T17:16:40.890 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:40 smithi116 systemd[1]: Stopping Ceph node-exporter.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:16:41.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:41 smithi116 bash[65980]: cephadm 2023-07-10T17:16:40.082997+0000 mgr.y (mgr.25390) 396 : cephadm 1 Upgrade: Updating node-exporter.a (1/2) 2023-07-10T17:16:41.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:41 smithi116 bash[65980]: cephadm 2023-07-10T17:16:40.083498+0000 mgr.y (mgr.25390) 397 : cephadm 1 Deploying daemon node-exporter.a on smithi116 2023-07-10T17:16:41.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:41 smithi116 bash[65980]: audit 2023-07-10T17:16:40.122770+0000 mon.a (mon.0) 990 : audit 1 from='client.? 172.21.15.116:0/1148106' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3104229546"}]': finished 2023-07-10T17:16:41.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:41 smithi116 bash[65980]: cluster 2023-07-10T17:16:40.122860+0000 mon.a (mon.0) 991 : cluster 0 osdmap e141: 8 total, 8 up, 8 in 2023-07-10T17:16:41.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:41 smithi116 bash[65980]: audit 2023-07-10T17:16:40.538229+0000 mon.c (mon.1) 33 : audit 1 from='client.? 172.21.15.116:0/2122772667' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: dispatch 2023-07-10T17:16:41.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:41 smithi116 bash[65980]: audit 2023-07-10T17:16:40.538915+0000 mon.a (mon.0) 992 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: dispatch 2023-07-10T17:16:41.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:41 smithi116 bash[63900]: cephadm 2023-07-10T17:16:40.082997+0000 mgr.y (mgr.25390) 396 : cephadm 1 Upgrade: Updating node-exporter.a (1/2) 2023-07-10T17:16:41.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:41 smithi116 bash[63900]: cephadm 2023-07-10T17:16:40.083498+0000 mgr.y (mgr.25390) 397 : cephadm 1 Deploying daemon node-exporter.a on smithi116 2023-07-10T17:16:41.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:41 smithi116 bash[63900]: audit 2023-07-10T17:16:40.122770+0000 mon.a (mon.0) 990 : audit 1 from='client.? 172.21.15.116:0/1148106' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3104229546"}]': finished 2023-07-10T17:16:41.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:41 smithi116 bash[63900]: cluster 2023-07-10T17:16:40.122860+0000 mon.a (mon.0) 991 : cluster 0 osdmap e141: 8 total, 8 up, 8 in 2023-07-10T17:16:41.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:41 smithi116 bash[63900]: audit 2023-07-10T17:16:40.538229+0000 mon.c (mon.1) 33 : audit 1 from='client.? 172.21.15.116:0/2122772667' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: dispatch 2023-07-10T17:16:41.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:41 smithi116 bash[63900]: audit 2023-07-10T17:16:40.538915+0000 mon.a (mon.0) 992 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: dispatch 2023-07-10T17:16:41.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:41 smithi192 bash[50362]: cephadm 2023-07-10T17:16:40.082997+0000 mgr.y (mgr.25390) 396 : cephadm 1 Upgrade: Updating node-exporter.a (1/2) 2023-07-10T17:16:41.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:41 smithi192 bash[50362]: cephadm 2023-07-10T17:16:40.083498+0000 mgr.y (mgr.25390) 397 : cephadm 1 Deploying daemon node-exporter.a on smithi116 2023-07-10T17:16:41.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:41 smithi192 bash[50362]: audit 2023-07-10T17:16:40.122770+0000 mon.a (mon.0) 990 : audit 1 from='client.? 172.21.15.116:0/1148106' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3104229546"}]': finished 2023-07-10T17:16:41.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:41 smithi192 bash[50362]: cluster 2023-07-10T17:16:40.122860+0000 mon.a (mon.0) 991 : cluster 0 osdmap e141: 8 total, 8 up, 8 in 2023-07-10T17:16:41.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:41 smithi192 bash[50362]: audit 2023-07-10T17:16:40.538229+0000 mon.c (mon.1) 33 : audit 1 from='client.? 172.21.15.116:0/2122772667' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: dispatch 2023-07-10T17:16:41.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:41 smithi192 bash[50362]: audit 2023-07-10T17:16:40.538915+0000 mon.a (mon.0) 992 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]: dispatch 2023-07-10T17:16:41.830 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:41 smithi116 bash[85693]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-node-exporter.a 2023-07-10T17:16:41.830 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:41 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2023-07-10T17:16:41.830 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:41 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@node-exporter.a.service: Failed with result 'exit-code'. 2023-07-10T17:16:41.831 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:41 smithi116 systemd[1]: Stopped Ceph node-exporter.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:16:42.139 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:41 smithi116 systemd[1]: Started Ceph node-exporter.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:16:42.141 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:41 smithi116 bash[85849]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2023-07-10T17:16:42.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:42 smithi116 bash[63900]: cluster 2023-07-10T17:16:41.108348+0000 mgr.y (mgr.25390) 398 : cluster 0 pgmap v295: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-10T17:16:42.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:42 smithi116 bash[63900]: audit 2023-07-10T17:16:41.129430+0000 mon.a (mon.0) 993 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]': finished 2023-07-10T17:16:42.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:42 smithi116 bash[63900]: cluster 2023-07-10T17:16:41.129657+0000 mon.a (mon.0) 994 : cluster 0 osdmap e142: 8 total, 8 up, 8 in 2023-07-10T17:16:42.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:42 smithi116 bash[63900]: audit 2023-07-10T17:16:41.179645+0000 mgr.y (mgr.25390) 399 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:16:42.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:42 smithi116 bash[63900]: audit 2023-07-10T17:16:41.863443+0000 mon.a (mon.0) 995 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:42.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:42 smithi116 bash[63900]: audit 2023-07-10T17:16:41.870711+0000 mon.a (mon.0) 996 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:42.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:42 smithi116 bash[65980]: cluster 2023-07-10T17:16:41.108348+0000 mgr.y (mgr.25390) 398 : cluster 0 pgmap v295: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-10T17:16:42.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:42 smithi116 bash[65980]: audit 2023-07-10T17:16:41.129430+0000 mon.a (mon.0) 993 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]': finished 2023-07-10T17:16:42.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:42 smithi116 bash[65980]: cluster 2023-07-10T17:16:41.129657+0000 mon.a (mon.0) 994 : cluster 0 osdmap e142: 8 total, 8 up, 8 in 2023-07-10T17:16:42.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:42 smithi116 bash[65980]: audit 2023-07-10T17:16:41.179645+0000 mgr.y (mgr.25390) 399 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:16:42.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:42 smithi116 bash[65980]: audit 2023-07-10T17:16:41.863443+0000 mon.a (mon.0) 995 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:42.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:42 smithi116 bash[65980]: audit 2023-07-10T17:16:41.870711+0000 mon.a (mon.0) 996 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:42.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:42 smithi192 bash[50362]: cluster 2023-07-10T17:16:41.108348+0000 mgr.y (mgr.25390) 398 : cluster 0 pgmap v295: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-10T17:16:42.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:42 smithi192 bash[50362]: audit 2023-07-10T17:16:41.129430+0000 mon.a (mon.0) 993 : audit 1 from='client.? ' entity='client.iscsi.foo.smithi116.fkthyl' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.116:0/3507856810"}]': finished 2023-07-10T17:16:42.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:42 smithi192 bash[50362]: cluster 2023-07-10T17:16:41.129657+0000 mon.a (mon.0) 994 : cluster 0 osdmap e142: 8 total, 8 up, 8 in 2023-07-10T17:16:42.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:42 smithi192 bash[50362]: audit 2023-07-10T17:16:41.179645+0000 mgr.y (mgr.25390) 399 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:16:42.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:42 smithi192 bash[50362]: audit 2023-07-10T17:16:41.863443+0000 mon.a (mon.0) 995 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:42.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:42 smithi192 bash[50362]: audit 2023-07-10T17:16:41.870711+0000 mon.a (mon.0) 996 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:43.039 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:42 smithi116 bash[85849]: v1.5.0: Pulling from prometheus/node-exporter 2023-07-10T17:16:43.394 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:43 smithi116 bash[85849]: 22b70bddd3ac: Pulling fs layer 2023-07-10T17:16:43.394 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:43 smithi116 bash[85849]: 5c12815fee55: Pulling fs layer 2023-07-10T17:16:43.395 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:43 smithi116 bash[85849]: c0e87333d380: Pulling fs layer 2023-07-10T17:16:43.728 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:43 smithi116 bash[65980]: cluster 2023-07-10T17:16:43.109711+0000 mgr.y (mgr.25390) 400 : cluster 0 pgmap v297: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 409 B/s wr, 6 op/s 2023-07-10T17:16:43.728 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:43 smithi116 bash[85849]: 5c12815fee55: Verifying Checksum 2023-07-10T17:16:43.729 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:43 smithi116 bash[85849]: 5c12815fee55: Download complete 2023-07-10T17:16:43.729 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:43 smithi116 bash[85849]: 22b70bddd3ac: Download complete 2023-07-10T17:16:43.729 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:43 smithi116 bash[63900]: cluster 2023-07-10T17:16:43.109711+0000 mgr.y (mgr.25390) 400 : cluster 0 pgmap v297: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 409 B/s wr, 6 op/s 2023-07-10T17:16:43.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:43 smithi192 bash[50362]: cluster 2023-07-10T17:16:43.109711+0000 mgr.y (mgr.25390) 400 : cluster 0 pgmap v297: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 409 B/s wr, 6 op/s 2023-07-10T17:16:44.144 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:43 smithi116 bash[85849]: 22b70bddd3ac: Pull complete 2023-07-10T17:16:44.144 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:43 smithi116 bash[85849]: c0e87333d380: Download complete 2023-07-10T17:16:44.574 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:44 smithi116 bash[85849]: 5c12815fee55: Pull complete 2023-07-10T17:16:44.887 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:44 smithi192 systemd[1]: Stopping Ceph node-exporter.b for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:16:44.894 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:44 smithi116 bash[85849]: c0e87333d380: Pull complete 2023-07-10T17:16:44.894 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:44 smithi116 bash[85849]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2023-07-10T17:16:44.894 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:44 smithi116 bash[85849]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2023-07-10T17:16:45.887 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:45 smithi192 bash[64741]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-node-exporter.b 2023-07-10T17:16:45.887 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:45 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2023-07-10T17:16:45.888 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:45 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@node-exporter.b.service: Failed with result 'exit-code'. 2023-07-10T17:16:45.888 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:45 smithi192 systemd[1]: Stopped Ceph node-exporter.b for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:16:46.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:46 smithi192 bash[50362]: cephadm 2023-07-10T17:16:44.320887+0000 mgr.y (mgr.25390) 401 : cephadm 1 Upgrade: Updating node-exporter.b (2/2) 2023-07-10T17:16:46.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:46 smithi192 bash[50362]: cephadm 2023-07-10T17:16:44.321560+0000 mgr.y (mgr.25390) 402 : cephadm 1 Deploying daemon node-exporter.b on smithi192 2023-07-10T17:16:46.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:46 smithi192 bash[50362]: cluster 2023-07-10T17:16:45.110865+0000 mgr.y (mgr.25390) 403 : cluster 0 pgmap v298: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 341 B/s wr, 5 op/s 2023-07-10T17:16:46.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:46 smithi192 bash[50362]: audit 2023-07-10T17:16:46.012886+0000 mon.a (mon.0) 997 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:46.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:46 smithi192 bash[50362]: audit 2023-07-10T17:16:46.023259+0000 mon.a (mon.0) 998 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:46.389 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:45 smithi192 systemd[1]: Started Ceph node-exporter.b for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:16:46.389 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:46 smithi192 bash[64901]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2023-07-10T17:16:46.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[63900]: cephadm 2023-07-10T17:16:44.320887+0000 mgr.y (mgr.25390) 401 : cephadm 1 Upgrade: Updating node-exporter.b (2/2) 2023-07-10T17:16:46.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[63900]: cephadm 2023-07-10T17:16:44.321560+0000 mgr.y (mgr.25390) 402 : cephadm 1 Deploying daemon node-exporter.b on smithi192 2023-07-10T17:16:46.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[63900]: cluster 2023-07-10T17:16:45.110865+0000 mgr.y (mgr.25390) 403 : cluster 0 pgmap v298: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 341 B/s wr, 5 op/s 2023-07-10T17:16:46.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[63900]: audit 2023-07-10T17:16:46.012886+0000 mon.a (mon.0) 997 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:46.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[63900]: audit 2023-07-10T17:16:46.023259+0000 mon.a (mon.0) 998 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:46.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[65980]: cephadm 2023-07-10T17:16:44.320887+0000 mgr.y (mgr.25390) 401 : cephadm 1 Upgrade: Updating node-exporter.b (2/2) 2023-07-10T17:16:46.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[65980]: cephadm 2023-07-10T17:16:44.321560+0000 mgr.y (mgr.25390) 402 : cephadm 1 Deploying daemon node-exporter.b on smithi192 2023-07-10T17:16:46.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[65980]: cluster 2023-07-10T17:16:45.110865+0000 mgr.y (mgr.25390) 403 : cluster 0 pgmap v298: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 341 B/s wr, 5 op/s 2023-07-10T17:16:46.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[65980]: audit 2023-07-10T17:16:46.012886+0000 mon.a (mon.0) 997 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:46.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[65980]: audit 2023-07-10T17:16:46.023259+0000 mon.a (mon.0) 998 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:46.647 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.255Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2023-07-10T17:16:46.647 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.255Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2023-07-10T17:16:46.647 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.255Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2023-07-10T17:16:46.647 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.256Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2023-07-10T17:16:46.647 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.256Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2023-07-10T17:16:46.647 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.256Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2023-07-10T17:16:46.647 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2023-07-10T17:16:46.648 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=arp 2023-07-10T17:16:46.648 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=bcache 2023-07-10T17:16:46.648 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=bonding 2023-07-10T17:16:46.648 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=btrfs 2023-07-10T17:16:46.648 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=conntrack 2023-07-10T17:16:46.648 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=cpu 2023-07-10T17:16:46.648 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=cpufreq 2023-07-10T17:16:46.649 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=diskstats 2023-07-10T17:16:46.649 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=dmi 2023-07-10T17:16:46.649 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=edac 2023-07-10T17:16:46.649 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=entropy 2023-07-10T17:16:46.649 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=fibrechannel 2023-07-10T17:16:46.651 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=filefd 2023-07-10T17:16:46.652 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=filesystem 2023-07-10T17:16:46.652 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=hwmon 2023-07-10T17:16:46.653 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=infiniband 2023-07-10T17:16:46.653 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=ipvs 2023-07-10T17:16:46.654 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=loadavg 2023-07-10T17:16:46.654 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=mdadm 2023-07-10T17:16:46.654 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=meminfo 2023-07-10T17:16:46.655 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=netclass 2023-07-10T17:16:46.655 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=netdev 2023-07-10T17:16:46.655 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=netstat 2023-07-10T17:16:46.655 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=nfs 2023-07-10T17:16:46.656 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=nfsd 2023-07-10T17:16:46.656 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=nvme 2023-07-10T17:16:46.656 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=os 2023-07-10T17:16:46.656 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2023-07-10T17:16:46.657 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=pressure 2023-07-10T17:16:46.657 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=rapl 2023-07-10T17:16:46.657 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=schedstat 2023-07-10T17:16:46.658 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=selinux 2023-07-10T17:16:46.658 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=sockstat 2023-07-10T17:16:46.658 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=softnet 2023-07-10T17:16:46.658 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=stat 2023-07-10T17:16:46.659 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=tapestats 2023-07-10T17:16:46.659 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=textfile 2023-07-10T17:16:46.659 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=thermal_zone 2023-07-10T17:16:46.660 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=time 2023-07-10T17:16:46.660 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=udp_queues 2023-07-10T17:16:46.660 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=uname 2023-07-10T17:16:46.661 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=vmstat 2023-07-10T17:16:46.666 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=xfs 2023-07-10T17:16:46.666 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=node_exporter.go:117 level=info collector=zfs 2023-07-10T17:16:46.667 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2023-07-10T17:16:46.667 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:16:46 smithi116 bash[85849]: ts=2023-07-10T17:16:46.257Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2023-07-10T17:16:46.861 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:46 smithi192 bash[64901]: v1.5.0: Pulling from prometheus/node-exporter 2023-07-10T17:16:47.137 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:46 smithi192 bash[64901]: 22b70bddd3ac: Pulling fs layer 2023-07-10T17:16:47.137 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:46 smithi192 bash[64901]: 5c12815fee55: Pulling fs layer 2023-07-10T17:16:47.137 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:46 smithi192 bash[64901]: c0e87333d380: Pulling fs layer 2023-07-10T17:16:47.511 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:47 smithi192 bash[64901]: 22b70bddd3ac: Verifying Checksum 2023-07-10T17:16:47.512 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:47 smithi192 bash[64901]: 22b70bddd3ac: Download complete 2023-07-10T17:16:47.512 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:47 smithi192 bash[64901]: 5c12815fee55: Verifying Checksum 2023-07-10T17:16:47.512 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:47 smithi192 bash[64901]: 5c12815fee55: Download complete 2023-07-10T17:16:47.512 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:47 smithi192 bash[64901]: 22b70bddd3ac: Pull complete 2023-07-10T17:16:47.833 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:47 smithi192 bash[64901]: c0e87333d380: Verifying Checksum 2023-07-10T17:16:47.834 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:47 smithi192 bash[64901]: c0e87333d380: Download complete 2023-07-10T17:16:48.137 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:47 smithi192 bash[64901]: 5c12815fee55: Pull complete 2023-07-10T17:16:48.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:48 smithi192 bash[50362]: cluster 2023-07-10T17:16:47.112254+0000 mgr.y (mgr.25390) 404 : cluster 0 pgmap v299: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 256 B/s wr, 4 op/s 2023-07-10T17:16:48.440 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:48 smithi192 bash[64901]: c0e87333d380: Pull complete 2023-07-10T17:16:48.440 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:48 smithi192 bash[64901]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2023-07-10T17:16:48.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:48 smithi116 bash[63900]: cluster 2023-07-10T17:16:47.112254+0000 mgr.y (mgr.25390) 404 : cluster 0 pgmap v299: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 256 B/s wr, 4 op/s 2023-07-10T17:16:48.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:48 smithi116 bash[65980]: cluster 2023-07-10T17:16:47.112254+0000 mgr.y (mgr.25390) 404 : cluster 0 pgmap v299: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 256 B/s wr, 4 op/s 2023-07-10T17:16:48.888 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:48 smithi192 bash[64901]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2023-07-10T17:16:50.137 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.885Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2023-07-10T17:16:50.138 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.886Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2023-07-10T17:16:50.138 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.886Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2023-07-10T17:16:50.138 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.886Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2023-07-10T17:16:50.138 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.887Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2023-07-10T17:16:50.138 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2023-07-10T17:16:50.138 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2023-07-10T17:16:50.139 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=arp 2023-07-10T17:16:50.139 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=bcache 2023-07-10T17:16:50.139 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=bonding 2023-07-10T17:16:50.139 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=btrfs 2023-07-10T17:16:50.139 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=conntrack 2023-07-10T17:16:50.140 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=cpu 2023-07-10T17:16:50.140 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=cpufreq 2023-07-10T17:16:50.140 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=diskstats 2023-07-10T17:16:50.140 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=dmi 2023-07-10T17:16:50.141 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=edac 2023-07-10T17:16:50.141 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=entropy 2023-07-10T17:16:50.141 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=fibrechannel 2023-07-10T17:16:50.141 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=filefd 2023-07-10T17:16:50.142 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=filesystem 2023-07-10T17:16:50.143 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=hwmon 2023-07-10T17:16:50.143 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=infiniband 2023-07-10T17:16:50.143 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=ipvs 2023-07-10T17:16:50.144 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=loadavg 2023-07-10T17:16:50.145 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=mdadm 2023-07-10T17:16:50.145 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=meminfo 2023-07-10T17:16:50.146 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=netclass 2023-07-10T17:16:50.146 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.888Z caller=node_exporter.go:117 level=info collector=netdev 2023-07-10T17:16:50.147 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=netstat 2023-07-10T17:16:50.147 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=nfs 2023-07-10T17:16:50.147 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=nfsd 2023-07-10T17:16:50.147 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=nvme 2023-07-10T17:16:50.148 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=os 2023-07-10T17:16:50.148 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2023-07-10T17:16:50.148 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=pressure 2023-07-10T17:16:50.149 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=rapl 2023-07-10T17:16:50.149 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=schedstat 2023-07-10T17:16:50.149 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=selinux 2023-07-10T17:16:50.149 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=sockstat 2023-07-10T17:16:50.150 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=softnet 2023-07-10T17:16:50.150 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=stat 2023-07-10T17:16:50.150 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=tapestats 2023-07-10T17:16:50.150 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=textfile 2023-07-10T17:16:50.150 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=thermal_zone 2023-07-10T17:16:50.151 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=time 2023-07-10T17:16:50.151 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=udp_queues 2023-07-10T17:16:50.151 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=uname 2023-07-10T17:16:50.152 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=vmstat 2023-07-10T17:16:50.152 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=xfs 2023-07-10T17:16:50.152 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.889Z caller=node_exporter.go:117 level=info collector=zfs 2023-07-10T17:16:50.153 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.890Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2023-07-10T17:16:50.153 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:16:49 smithi192 bash[64901]: ts=2023-07-10T17:16:49.890Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2023-07-10T17:16:50.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:50 smithi192 bash[50362]: cluster 2023-07-10T17:16:49.113212+0000 mgr.y (mgr.25390) 405 : cluster 0 pgmap v300: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 113 B/s wr, 3 op/s 2023-07-10T17:16:50.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:50 smithi116 bash[63900]: cluster 2023-07-10T17:16:49.113212+0000 mgr.y (mgr.25390) 405 : cluster 0 pgmap v300: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 113 B/s wr, 3 op/s 2023-07-10T17:16:50.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:50 smithi116 bash[65980]: cluster 2023-07-10T17:16:49.113212+0000 mgr.y (mgr.25390) 405 : cluster 0 pgmap v300: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 113 B/s wr, 3 op/s 2023-07-10T17:16:52.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:52 smithi192 bash[50362]: cluster 2023-07-10T17:16:51.114225+0000 mgr.y (mgr.25390) 406 : cluster 0 pgmap v301: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 1 op/s 2023-07-10T17:16:52.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:52 smithi192 bash[50362]: audit 2023-07-10T17:16:51.187702+0000 mgr.y (mgr.25390) 407 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:16:52.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:52 smithi116 bash[63900]: cluster 2023-07-10T17:16:51.114225+0000 mgr.y (mgr.25390) 406 : cluster 0 pgmap v301: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 1 op/s 2023-07-10T17:16:52.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:52 smithi116 bash[63900]: audit 2023-07-10T17:16:51.187702+0000 mgr.y (mgr.25390) 407 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:16:52.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:52 smithi116 bash[65980]: cluster 2023-07-10T17:16:51.114225+0000 mgr.y (mgr.25390) 406 : cluster 0 pgmap v301: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 1 op/s 2023-07-10T17:16:52.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:52 smithi116 bash[65980]: audit 2023-07-10T17:16:51.187702+0000 mgr.y (mgr.25390) 407 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:16:53.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:53 smithi192 bash[50362]: audit 2023-07-10T17:16:52.434340+0000 mon.a (mon.0) 999 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:53 smithi192 bash[50362]: audit 2023-07-10T17:16:52.444173+0000 mon.a (mon.0) 1000 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:53 smithi192 bash[50362]: cluster 2023-07-10T17:16:53.115377+0000 mgr.y (mgr.25390) 408 : cluster 0 pgmap v302: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2023-07-10T17:16:53.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:53 smithi192 bash[50362]: audit 2023-07-10T17:16:53.138142+0000 mon.a (mon.0) 1001 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:53 smithi192 bash[50362]: audit 2023-07-10T17:16:53.146772+0000 mon.a (mon.0) 1002 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:53 smithi192 bash[50362]: audit 2023-07-10T17:16:53.242825+0000 mon.a (mon.0) 1003 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:53 smithi192 bash[50362]: audit 2023-07-10T17:16:53.253477+0000 mon.a (mon.0) 1004 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:53 smithi116 bash[63900]: audit 2023-07-10T17:16:52.434340+0000 mon.a (mon.0) 999 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:53 smithi116 bash[63900]: audit 2023-07-10T17:16:52.444173+0000 mon.a (mon.0) 1000 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:53 smithi116 bash[63900]: cluster 2023-07-10T17:16:53.115377+0000 mgr.y (mgr.25390) 408 : cluster 0 pgmap v302: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2023-07-10T17:16:53.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:53 smithi116 bash[63900]: audit 2023-07-10T17:16:53.138142+0000 mon.a (mon.0) 1001 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:53 smithi116 bash[63900]: audit 2023-07-10T17:16:53.146772+0000 mon.a (mon.0) 1002 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:53 smithi116 bash[63900]: audit 2023-07-10T17:16:53.242825+0000 mon.a (mon.0) 1003 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:53 smithi116 bash[63900]: audit 2023-07-10T17:16:53.253477+0000 mon.a (mon.0) 1004 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:53 smithi116 bash[65980]: audit 2023-07-10T17:16:52.434340+0000 mon.a (mon.0) 999 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:53 smithi116 bash[65980]: audit 2023-07-10T17:16:52.444173+0000 mon.a (mon.0) 1000 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:53 smithi116 bash[65980]: cluster 2023-07-10T17:16:53.115377+0000 mgr.y (mgr.25390) 408 : cluster 0 pgmap v302: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2023-07-10T17:16:53.896 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:53 smithi116 bash[65980]: audit 2023-07-10T17:16:53.138142+0000 mon.a (mon.0) 1001 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:53 smithi116 bash[65980]: audit 2023-07-10T17:16:53.146772+0000 mon.a (mon.0) 1002 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:53 smithi116 bash[65980]: audit 2023-07-10T17:16:53.242825+0000 mon.a (mon.0) 1003 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:53.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:53 smithi116 bash[65980]: audit 2023-07-10T17:16:53.253477+0000 mon.a (mon.0) 1004 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:55.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:54 smithi192 bash[50362]: audit 2023-07-10T17:16:53.949617+0000 mon.a (mon.0) 1005 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:55.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:54 smithi192 bash[50362]: audit 2023-07-10T17:16:53.962516+0000 mon.a (mon.0) 1006 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:55.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:54 smithi116 bash[65980]: audit 2023-07-10T17:16:53.949617+0000 mon.a (mon.0) 1005 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:55.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:54 smithi116 bash[65980]: audit 2023-07-10T17:16:53.962516+0000 mon.a (mon.0) 1006 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:55.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:54 smithi116 bash[63900]: audit 2023-07-10T17:16:53.949617+0000 mon.a (mon.0) 1005 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:55.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:54 smithi116 bash[63900]: audit 2023-07-10T17:16:53.962516+0000 mon.a (mon.0) 1006 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:16:56.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:55 smithi192 bash[50362]: cluster 2023-07-10T17:16:55.116443+0000 mgr.y (mgr.25390) 409 : cluster 0 pgmap v303: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:16:56.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:55 smithi116 bash[63900]: cluster 2023-07-10T17:16:55.116443+0000 mgr.y (mgr.25390) 409 : cluster 0 pgmap v303: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:16:56.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:55 smithi116 bash[65980]: cluster 2023-07-10T17:16:55.116443+0000 mgr.y (mgr.25390) 409 : cluster 0 pgmap v303: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:16:58.211 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:16:58.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:58 smithi192 bash[50362]: cluster 2023-07-10T17:16:57.117834+0000 mgr.y (mgr.25390) 410 : cluster 0 pgmap v304: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:16:58.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:58 smithi116 bash[65980]: cluster 2023-07-10T17:16:57.117834+0000 mgr.y (mgr.25390) 410 : cluster 0 pgmap v304: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:16:58.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:58 smithi116 bash[63900]: cluster 2023-07-10T17:16:57.117834+0000 mgr.y (mgr.25390) 410 : cluster 0 pgmap v304: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:16:58.946 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:16:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (10m) 6s ago 16m 11.0M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:16:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (8m) 5s ago 16m 26.3M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:16:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 running (39s) 6s ago 13m 47.5M - 3.6 17e8f0d79247 fbdecd8618b7 2023-07-10T17:16:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (7m) 5s ago 20m 431M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:16:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (8m) 6s ago 22m 483M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:16:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (7m) 6s ago 22m 82.4M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:16:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (6m) 5s ago 21m 62.1M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 74a082a2e795 2023-07-10T17:16:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (6m) 6s ago 21m 63.7M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:16:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (14s) 6s ago 16m 4591k - 1.5.0 0da6a335fe13 64b482421ffb 2023-07-10T17:16:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (10s) 5s ago 16m - - 1.5.0 0da6a335fe13 82b166142066 2023-07-10T17:16:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (5m) 6s ago 20m 63.2M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 0562f587fbca 2023-07-10T17:16:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (4m) 6s ago 19m 62.9M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 5b2a12a0bfc9 2023-07-10T17:16:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (4m) 6s ago 19m 55.8M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 88096bc0faa9 2023-07-10T17:16:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (3m) 6s ago 19m 60.2M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 61ae280c471b 2023-07-10T17:16:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (3m) 5s ago 18m 61.0M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 44d809c18118 2023-07-10T17:16:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (2m) 5s ago 18m 57.4M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 179270eb7044 2023-07-10T17:16:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (2m) 5s ago 17m 55.0M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 b680bfbe9a0c 2023-07-10T17:16:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (104s) 5s ago 17m 56.7M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 bd3788803e68 2023-07-10T17:16:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 error 5s ago 16m - - 2023-07-10T17:16:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (81s) 6s ago 14m 84.0M - 18.1.2-30-g8507faa7 17e8f0d79247 ef72a4e0ae29 2023-07-10T17:16:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (68s) 5s ago 14m 84.1M - 18.1.2-30-g8507faa7 17e8f0d79247 59b0f0b41f44 2023-07-10T17:16:58.951 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (74s) 6s ago 14m 84.0M - 18.1.2-30-g8507faa7 17e8f0d79247 42f4f263b75e 2023-07-10T17:16:58.951 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (61s) 5s ago 14m 83.8M - 18.1.2-30-g8507faa7 17e8f0d79247 2a257a88c4e9 2023-07-10T17:16:59.388 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:16:59.389 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:16:59.389 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:16:59.390 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:16:59.390 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:16:59.390 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:16:59.390 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:16:59.391 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:16:59.391 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 8 2023-07-10T17:16:59.391 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:16:59.391 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:16:59.391 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 4 2023-07-10T17:16:59.392 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:16:59.392 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:16:59.392 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 17 2023-07-10T17:16:59.392 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:16:59.393 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:16:59.486 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:16:59 smithi116 bash[65980]: audit 2023-07-10T17:16:58.191442+0000 mgr.y (mgr.25390) 411 : audit 0 from='client.35091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:16:59.487 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:16:59 smithi116 bash[63900]: audit 2023-07-10T17:16:58.191442+0000 mgr.y (mgr.25390) 411 : audit 0 from='client.35091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:16:59.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:16:59 smithi192 bash[50362]: audit 2023-07-10T17:16:58.191442+0000 mgr.y (mgr.25390) 411 : audit 0 from='client.35091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:16:59.757 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:16:59.758 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:16:59.758 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:16:59.758 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:16:59.758 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:16:59.758 INFO:teuthology.orchestra.run.smithi116.stdout: "iscsi", 2023-07-10T17:16:59.759 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw", 2023-07-10T17:16:59.759 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2023-07-10T17:16:59.759 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr", 2023-07-10T17:16:59.759 INFO:teuthology.orchestra.run.smithi116.stdout: "osd" 2023-07-10T17:16:59.759 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:16:59.759 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "18/23 daemons upgraded", 2023-07-10T17:16:59.759 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading node-exporter daemons", 2023-07-10T17:16:59.760 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:16:59.761 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:17:00.196 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2023-07-10T17:17:00.197 INFO:teuthology.orchestra.run.smithi116.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2023-07-10T17:17:00.197 INFO:teuthology.orchestra.run.smithi116.stdout: daemon prometheus.a on smithi192 is in error state 2023-07-10T17:17:00.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:00 smithi192 bash[50362]: audit 2023-07-10T17:16:58.560603+0000 mgr.y (mgr.25390) 412 : audit 0 from='client.35097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:00.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:00 smithi192 bash[50362]: audit 2023-07-10T17:16:58.919278+0000 mgr.y (mgr.25390) 413 : audit 0 from='client.35103 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:00.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:00 smithi192 bash[50362]: cluster 2023-07-10T17:16:59.118623+0000 mgr.y (mgr.25390) 414 : cluster 0 pgmap v305: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:00.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:00 smithi192 bash[50362]: audit 2023-07-10T17:16:59.384239+0000 mon.c (mon.1) 34 : audit 0 from='client.44971 172.21.15.116:0/2121772613' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:00.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:00 smithi116 bash[63900]: audit 2023-07-10T17:16:58.560603+0000 mgr.y (mgr.25390) 412 : audit 0 from='client.35097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:00.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:00 smithi116 bash[63900]: audit 2023-07-10T17:16:58.919278+0000 mgr.y (mgr.25390) 413 : audit 0 from='client.35103 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:00.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:00 smithi116 bash[63900]: cluster 2023-07-10T17:16:59.118623+0000 mgr.y (mgr.25390) 414 : cluster 0 pgmap v305: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:00.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:00 smithi116 bash[63900]: audit 2023-07-10T17:16:59.384239+0000 mon.c (mon.1) 34 : audit 0 from='client.44971 172.21.15.116:0/2121772613' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:00.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:00 smithi116 bash[65980]: audit 2023-07-10T17:16:58.560603+0000 mgr.y (mgr.25390) 412 : audit 0 from='client.35097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:00.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:00 smithi116 bash[65980]: audit 2023-07-10T17:16:58.919278+0000 mgr.y (mgr.25390) 413 : audit 0 from='client.35103 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:00.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:00 smithi116 bash[65980]: cluster 2023-07-10T17:16:59.118623+0000 mgr.y (mgr.25390) 414 : cluster 0 pgmap v305: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:00.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:00 smithi116 bash[65980]: audit 2023-07-10T17:16:59.384239+0000 mon.c (mon.1) 34 : audit 0 from='client.44971 172.21.15.116:0/2121772613' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:01.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:01 smithi192 bash[50362]: audit 2023-07-10T17:16:59.752268+0000 mgr.y (mgr.25390) 415 : audit 0 from='client.44977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:01.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:01 smithi192 bash[50362]: audit 2023-07-10T17:17:00.191554+0000 mon.a (mon.0) 1007 : audit 0 from='client.35121 172.21.15.116:0/776355522' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:17:01.591 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:01 smithi116 bash[63900]: audit 2023-07-10T17:16:59.752268+0000 mgr.y (mgr.25390) 415 : audit 0 from='client.44977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:01.592 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:01 smithi116 bash[63900]: audit 2023-07-10T17:17:00.191554+0000 mon.a (mon.0) 1007 : audit 0 from='client.35121 172.21.15.116:0/776355522' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:17:01.592 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:01 smithi116 bash[65980]: audit 2023-07-10T17:16:59.752268+0000 mgr.y (mgr.25390) 415 : audit 0 from='client.44977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:01.592 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:01 smithi116 bash[65980]: audit 2023-07-10T17:17:00.191554+0000 mon.a (mon.0) 1007 : audit 0 from='client.35121 172.21.15.116:0/776355522' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:17:02.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:02 smithi116 bash[65980]: cluster 2023-07-10T17:17:01.120052+0000 mgr.y (mgr.25390) 416 : cluster 0 pgmap v306: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:02.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:02 smithi116 bash[65980]: audit 2023-07-10T17:17:01.198486+0000 mgr.y (mgr.25390) 417 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:02.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:02 smithi116 bash[63900]: cluster 2023-07-10T17:17:01.120052+0000 mgr.y (mgr.25390) 416 : cluster 0 pgmap v306: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:02.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:02 smithi116 bash[63900]: audit 2023-07-10T17:17:01.198486+0000 mgr.y (mgr.25390) 417 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:02.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:02 smithi192 bash[50362]: cluster 2023-07-10T17:17:01.120052+0000 mgr.y (mgr.25390) 416 : cluster 0 pgmap v306: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:02.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:02 smithi192 bash[50362]: audit 2023-07-10T17:17:01.198486+0000 mgr.y (mgr.25390) 417 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:04.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:02.961022+0000 mon.a (mon.0) 1008 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:04.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:02.971645+0000 mon.a (mon.0) 1009 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:04.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: cluster 2023-07-10T17:17:03.121274+0000 mgr.y (mgr.25390) 418 : cluster 0 pgmap v307: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:04.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.414204+0000 mon.a (mon.0) 1010 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:04.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.424205+0000 mon.a (mon.0) 1011 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:04.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.426258+0000 mon.a (mon.0) 1012 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:17:04.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.427726+0000 mon.a (mon.0) 1013 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:17:04.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.438837+0000 mon.a (mon.0) 1014 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:04.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.527330+0000 mon.a (mon.0) 1015 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:17:04.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.532369+0000 mon.a (mon.0) 1016 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.534959+0000 mon.a (mon.0) 1017 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.536895+0000 mon.a (mon.0) 1018 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.542549+0000 mon.a (mon.0) 1019 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.549036+0000 mon.a (mon.0) 1020 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.552361+0000 mon.a (mon.0) 1021 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.554300+0000 mon.a (mon.0) 1022 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.556383+0000 mon.a (mon.0) 1023 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.558473+0000 mon.a (mon.0) 1024 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.560672+0000 mon.a (mon.0) 1025 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.562708+0000 mon.a (mon.0) 1026 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:03 smithi192 bash[50362]: audit 2023-07-10T17:17:03.565591+0000 mon.a (mon.0) 1027 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:02.961022+0000 mon.a (mon.0) 1008 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:04.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:02.971645+0000 mon.a (mon.0) 1009 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:04.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: cluster 2023-07-10T17:17:03.121274+0000 mgr.y (mgr.25390) 418 : cluster 0 pgmap v307: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:04.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.414204+0000 mon.a (mon.0) 1010 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:04.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.424205+0000 mon.a (mon.0) 1011 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:04.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.426258+0000 mon.a (mon.0) 1012 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:17:04.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.427726+0000 mon.a (mon.0) 1013 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:17:04.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.438837+0000 mon.a (mon.0) 1014 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:04.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.527330+0000 mon.a (mon.0) 1015 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:17:04.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.532369+0000 mon.a (mon.0) 1016 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.534959+0000 mon.a (mon.0) 1017 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.536895+0000 mon.a (mon.0) 1018 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.542549+0000 mon.a (mon.0) 1019 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.549036+0000 mon.a (mon.0) 1020 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.552361+0000 mon.a (mon.0) 1021 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.554300+0000 mon.a (mon.0) 1022 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.556383+0000 mon.a (mon.0) 1023 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.558473+0000 mon.a (mon.0) 1024 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.560672+0000 mon.a (mon.0) 1025 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.562708+0000 mon.a (mon.0) 1026 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[63900]: audit 2023-07-10T17:17:03.565591+0000 mon.a (mon.0) 1027 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:02.961022+0000 mon.a (mon.0) 1008 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:04.400 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:02.971645+0000 mon.a (mon.0) 1009 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:04.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: cluster 2023-07-10T17:17:03.121274+0000 mgr.y (mgr.25390) 418 : cluster 0 pgmap v307: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:04.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.414204+0000 mon.a (mon.0) 1010 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:04.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.424205+0000 mon.a (mon.0) 1011 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:04.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.426258+0000 mon.a (mon.0) 1012 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:17:04.401 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.427726+0000 mon.a (mon.0) 1013 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:17:04.402 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.438837+0000 mon.a (mon.0) 1014 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:04.402 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.527330+0000 mon.a (mon.0) 1015 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:17:04.402 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.532369+0000 mon.a (mon.0) 1016 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.402 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.534959+0000 mon.a (mon.0) 1017 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.402 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.536895+0000 mon.a (mon.0) 1018 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.402 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.542549+0000 mon.a (mon.0) 1019 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.402 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.549036+0000 mon.a (mon.0) 1020 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.403 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.552361+0000 mon.a (mon.0) 1021 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.403 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.554300+0000 mon.a (mon.0) 1022 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.403 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.556383+0000 mon.a (mon.0) 1023 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.403 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.558473+0000 mon.a (mon.0) 1024 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.403 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.560672+0000 mon.a (mon.0) 1025 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.403 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.562708+0000 mon.a (mon.0) 1026 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:04.403 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:03 smithi116 bash[65980]: audit 2023-07-10T17:17:03.565591+0000 mon.a (mon.0) 1027 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:05.274 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:04 smithi192 bash[50362]: cephadm 2023-07-10T17:17:03.429000+0000 mgr.y (mgr.25390) 419 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:17:05.274 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:04 smithi192 bash[50362]: cluster 2023-07-10T17:17:03.430191+0000 mgr.y (mgr.25390) 420 : cluster 0 pgmap v308: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:05.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:04 smithi116 bash[65980]: cephadm 2023-07-10T17:17:03.429000+0000 mgr.y (mgr.25390) 419 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:17:05.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:04 smithi116 bash[65980]: cluster 2023-07-10T17:17:03.430191+0000 mgr.y (mgr.25390) 420 : cluster 0 pgmap v308: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:05.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:04 smithi116 bash[63900]: cephadm 2023-07-10T17:17:03.429000+0000 mgr.y (mgr.25390) 419 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:17:05.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:04 smithi116 bash[63900]: cluster 2023-07-10T17:17:03.430191+0000 mgr.y (mgr.25390) 420 : cluster 0 pgmap v308: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:07.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:06 smithi192 bash[50362]: cluster 2023-07-10T17:17:05.431319+0000 mgr.y (mgr.25390) 421 : cluster 0 pgmap v309: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:07.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:06 smithi192 bash[50362]: cephadm 2023-07-10T17:17:05.788217+0000 mgr.y (mgr.25390) 422 : cephadm 1 Upgrade: Updating prometheus.a 2023-07-10T17:17:07.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:06 smithi192 bash[50362]: cephadm 2023-07-10T17:17:06.033438+0000 mgr.y (mgr.25390) 423 : cephadm 1 Deploying daemon prometheus.a on smithi192 2023-07-10T17:17:07.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:06 smithi116 bash[65980]: cluster 2023-07-10T17:17:05.431319+0000 mgr.y (mgr.25390) 421 : cluster 0 pgmap v309: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:07.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:06 smithi116 bash[65980]: cephadm 2023-07-10T17:17:05.788217+0000 mgr.y (mgr.25390) 422 : cephadm 1 Upgrade: Updating prometheus.a 2023-07-10T17:17:07.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:06 smithi116 bash[65980]: cephadm 2023-07-10T17:17:06.033438+0000 mgr.y (mgr.25390) 423 : cephadm 1 Deploying daemon prometheus.a on smithi192 2023-07-10T17:17:07.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:06 smithi116 bash[63900]: cluster 2023-07-10T17:17:05.431319+0000 mgr.y (mgr.25390) 421 : cluster 0 pgmap v309: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:07.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:06 smithi116 bash[63900]: cephadm 2023-07-10T17:17:05.788217+0000 mgr.y (mgr.25390) 422 : cephadm 1 Upgrade: Updating prometheus.a 2023-07-10T17:17:07.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:06 smithi116 bash[63900]: cephadm 2023-07-10T17:17:06.033438+0000 mgr.y (mgr.25390) 423 : cephadm 1 Deploying daemon prometheus.a on smithi192 2023-07-10T17:17:09.339 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:08 smithi192 bash[50362]: cluster 2023-07-10T17:17:07.432770+0000 mgr.y (mgr.25390) 424 : cluster 0 pgmap v310: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:09.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:08 smithi116 bash[65980]: cluster 2023-07-10T17:17:07.432770+0000 mgr.y (mgr.25390) 424 : cluster 0 pgmap v310: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:09.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:08 smithi116 bash[63900]: cluster 2023-07-10T17:17:07.432770+0000 mgr.y (mgr.25390) 424 : cluster 0 pgmap v310: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:10.625 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:10 smithi192 systemd[1]: Started Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:17:11.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:10 smithi192 bash[50362]: cluster 2023-07-10T17:17:09.433791+0000 mgr.y (mgr.25390) 425 : cluster 0 pgmap v311: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:11.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:10 smithi192 bash[50362]: audit 2023-07-10T17:17:10.664741+0000 mon.a (mon.0) 1028 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:11.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:10 smithi192 bash[50362]: audit 2023-07-10T17:17:10.673541+0000 mon.a (mon.0) 1029 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:11.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:10 smithi116 bash[63900]: cluster 2023-07-10T17:17:09.433791+0000 mgr.y (mgr.25390) 425 : cluster 0 pgmap v311: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:11.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:10 smithi116 bash[63900]: audit 2023-07-10T17:17:10.664741+0000 mon.a (mon.0) 1028 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:11.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:10 smithi116 bash[63900]: audit 2023-07-10T17:17:10.673541+0000 mon.a (mon.0) 1029 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:11.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:10 smithi116 bash[65980]: cluster 2023-07-10T17:17:09.433791+0000 mgr.y (mgr.25390) 425 : cluster 0 pgmap v311: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:11.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:10 smithi116 bash[65980]: audit 2023-07-10T17:17:10.664741+0000 mon.a (mon.0) 1028 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:11.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:10 smithi116 bash[65980]: audit 2023-07-10T17:17:10.673541+0000 mon.a (mon.0) 1029 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:12.004 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.875Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2023-07-10T17:17:12.005 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.875Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2023-07-10T17:17:12.005 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.875Z caller=main.go:570 level=info host_details="(Linux 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 smithi192 (none))" 2023-07-10T17:17:12.005 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.875Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2023-07-10T17:17:12.006 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.875Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-10T17:17:12.006 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.877Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2023-07-10T17:17:12.006 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.878Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2023-07-10T17:17:12.007 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.878Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/prometheus/lock 2023-07-10T17:17:12.007 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.879Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2023-07-10T17:17:12.008 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.879Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2023-07-10T17:17:12.008 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.885Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2023-07-10T17:17:12.008 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.885Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.934µs 2023-07-10T17:17:12.008 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.885Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2023-07-10T17:17:12.009 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.885Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=13 2023-07-10T17:17:12.009 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.900Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=13 2023-07-10T17:17:12.009 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.900Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=13 2023-07-10T17:17:12.009 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.901Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=13 2023-07-10T17:17:12.012 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.901Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=13 2023-07-10T17:17:12.012 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.901Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=13 2023-07-10T17:17:12.013 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.901Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=13 2023-07-10T17:17:12.013 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.902Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=7 maxSegment=13 2023-07-10T17:17:12.013 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.902Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=8 maxSegment=13 2023-07-10T17:17:12.013 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.902Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=9 maxSegment=13 2023-07-10T17:17:12.014 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.903Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=10 maxSegment=13 2023-07-10T17:17:12.014 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.903Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=11 maxSegment=13 2023-07-10T17:17:12.014 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.903Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=12 maxSegment=13 2023-07-10T17:17:12.014 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.903Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=13 maxSegment=13 2023-07-10T17:17:12.015 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.903Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=39.821µs wal_replay_duration=18.711865ms wbl_replay_duration=152ns total_replay_duration=18.776424ms 2023-07-10T17:17:12.015 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.905Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2023-07-10T17:17:12.015 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.905Z caller=main.go:1029 level=info msg="TSDB started" 2023-07-10T17:17:12.015 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.905Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-10T17:17:12.015 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.931Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=25.987118ms db_storage=1.289µs remote_storage=1.426µs web_handler=674ns query_engine=785ns scrape=356.993µs scrape_sd=95.116µs notify=20.409µs notify_sd=18.523µs rules=25.194553ms tracing=8.2µs 2023-07-10T17:17:12.015 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.931Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2023-07-10T17:17:12.016 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[66523]: ts=2023-07-10T17:17:11.931Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2023-07-10T17:17:12.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:11 smithi192 bash[50362]: audit 2023-07-10T17:17:11.207700+0000 mgr.y (mgr.25390) 426 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:12.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:11 smithi116 bash[63900]: audit 2023-07-10T17:17:11.207700+0000 mgr.y (mgr.25390) 426 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:12.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:11 smithi116 bash[65980]: audit 2023-07-10T17:17:11.207700+0000 mgr.y (mgr.25390) 426 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:13.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:13 smithi192 bash[50362]: cluster 2023-07-10T17:17:11.435237+0000 mgr.y (mgr.25390) 427 : cluster 0 pgmap v312: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:13.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:12 smithi116 bash[63900]: cluster 2023-07-10T17:17:11.435237+0000 mgr.y (mgr.25390) 427 : cluster 0 pgmap v312: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:13.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:13 smithi116 bash[65980]: cluster 2023-07-10T17:17:11.435237+0000 mgr.y (mgr.25390) 427 : cluster 0 pgmap v312: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:15.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:15 smithi192 bash[50362]: cluster 2023-07-10T17:17:13.436390+0000 mgr.y (mgr.25390) 428 : cluster 0 pgmap v313: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:15.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:15 smithi116 bash[63900]: cluster 2023-07-10T17:17:13.436390+0000 mgr.y (mgr.25390) 428 : cluster 0 pgmap v313: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:15.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:15 smithi116 bash[65980]: cluster 2023-07-10T17:17:13.436390+0000 mgr.y (mgr.25390) 428 : cluster 0 pgmap v313: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2023-07-10T17:17:17.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:17 smithi192 bash[50362]: cluster 2023-07-10T17:17:15.437642+0000 mgr.y (mgr.25390) 429 : cluster 0 pgmap v314: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:17.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:17 smithi192 bash[50362]: audit 2023-07-10T17:17:16.958919+0000 mon.a (mon.0) 1030 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:17.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:17 smithi192 bash[50362]: audit 2023-07-10T17:17:16.969398+0000 mon.a (mon.0) 1031 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:17.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:17 smithi116 bash[63900]: cluster 2023-07-10T17:17:15.437642+0000 mgr.y (mgr.25390) 429 : cluster 0 pgmap v314: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:17.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:17 smithi116 bash[63900]: audit 2023-07-10T17:17:16.958919+0000 mon.a (mon.0) 1030 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:17.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:17 smithi116 bash[63900]: audit 2023-07-10T17:17:16.969398+0000 mon.a (mon.0) 1031 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:17.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:17 smithi116 bash[65980]: cluster 2023-07-10T17:17:15.437642+0000 mgr.y (mgr.25390) 429 : cluster 0 pgmap v314: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:17.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:17 smithi116 bash[65980]: audit 2023-07-10T17:17:16.958919+0000 mon.a (mon.0) 1030 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:17.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:17 smithi116 bash[65980]: audit 2023-07-10T17:17:16.969398+0000 mon.a (mon.0) 1031 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:18.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:18 smithi192 bash[50362]: audit 2023-07-10T17:17:17.108787+0000 mon.a (mon.0) 1032 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:18.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:18 smithi192 bash[50362]: audit 2023-07-10T17:17:17.119706+0000 mon.a (mon.0) 1033 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:18.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:18 smithi192 bash[50362]: audit 2023-07-10T17:17:17.774148+0000 mon.a (mon.0) 1034 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:18.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:18 smithi192 bash[50362]: audit 2023-07-10T17:17:17.783401+0000 mon.a (mon.0) 1035 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:18.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:18 smithi116 bash[63900]: audit 2023-07-10T17:17:17.108787+0000 mon.a (mon.0) 1032 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:18.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:18 smithi116 bash[63900]: audit 2023-07-10T17:17:17.119706+0000 mon.a (mon.0) 1033 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:18.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:18 smithi116 bash[63900]: audit 2023-07-10T17:17:17.774148+0000 mon.a (mon.0) 1034 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:18.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:18 smithi116 bash[63900]: audit 2023-07-10T17:17:17.783401+0000 mon.a (mon.0) 1035 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:18.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:18 smithi116 bash[65980]: audit 2023-07-10T17:17:17.108787+0000 mon.a (mon.0) 1032 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:18.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:18 smithi116 bash[65980]: audit 2023-07-10T17:17:17.119706+0000 mon.a (mon.0) 1033 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:18.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:18 smithi116 bash[65980]: audit 2023-07-10T17:17:17.774148+0000 mon.a (mon.0) 1034 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:18.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:18 smithi116 bash[65980]: audit 2023-07-10T17:17:17.783401+0000 mon.a (mon.0) 1035 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:19.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:19 smithi192 bash[50362]: cluster 2023-07-10T17:17:17.439065+0000 mgr.y (mgr.25390) 430 : cluster 0 pgmap v315: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:19.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:19 smithi116 bash[63900]: cluster 2023-07-10T17:17:17.439065+0000 mgr.y (mgr.25390) 430 : cluster 0 pgmap v315: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:19.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:19 smithi116 bash[65980]: cluster 2023-07-10T17:17:17.439065+0000 mgr.y (mgr.25390) 430 : cluster 0 pgmap v315: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:21.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:21 smithi192 bash[50362]: cluster 2023-07-10T17:17:19.439975+0000 mgr.y (mgr.25390) 431 : cluster 0 pgmap v316: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:21.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:21 smithi116 bash[65980]: cluster 2023-07-10T17:17:19.439975+0000 mgr.y (mgr.25390) 431 : cluster 0 pgmap v316: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:21.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:21 smithi116 bash[63900]: cluster 2023-07-10T17:17:19.439975+0000 mgr.y (mgr.25390) 431 : cluster 0 pgmap v316: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:22.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:22 smithi116 bash[63900]: audit 2023-07-10T17:17:21.215882+0000 mgr.y (mgr.25390) 432 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:22.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:22 smithi116 bash[65980]: audit 2023-07-10T17:17:21.215882+0000 mgr.y (mgr.25390) 432 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:22.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:22 smithi192 bash[50362]: audit 2023-07-10T17:17:21.215882+0000 mgr.y (mgr.25390) 432 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:23.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:23 smithi116 bash[63900]: cluster 2023-07-10T17:17:21.441231+0000 mgr.y (mgr.25390) 433 : cluster 0 pgmap v317: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:23.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:23 smithi116 bash[65980]: cluster 2023-07-10T17:17:21.441231+0000 mgr.y (mgr.25390) 433 : cluster 0 pgmap v317: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:23.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:23 smithi192 bash[50362]: cluster 2023-07-10T17:17:21.441231+0000 mgr.y (mgr.25390) 433 : cluster 0 pgmap v317: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:25.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:25 smithi192 bash[50362]: cluster 2023-07-10T17:17:23.442445+0000 mgr.y (mgr.25390) 434 : cluster 0 pgmap v318: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:25.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:25 smithi116 bash[63900]: cluster 2023-07-10T17:17:23.442445+0000 mgr.y (mgr.25390) 434 : cluster 0 pgmap v318: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:25.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:25 smithi116 bash[65980]: cluster 2023-07-10T17:17:23.442445+0000 mgr.y (mgr.25390) 434 : cluster 0 pgmap v318: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:27.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:27 smithi192 bash[50362]: cluster 2023-07-10T17:17:25.443603+0000 mgr.y (mgr.25390) 435 : cluster 0 pgmap v319: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:27.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:27 smithi116 bash[65980]: cluster 2023-07-10T17:17:25.443603+0000 mgr.y (mgr.25390) 435 : cluster 0 pgmap v319: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:27.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:27 smithi116 bash[63900]: cluster 2023-07-10T17:17:25.443603+0000 mgr.y (mgr.25390) 435 : cluster 0 pgmap v319: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:28.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.286035+0000 mon.a (mon.0) 1036 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:28.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.296698+0000 mon.a (mon.0) 1037 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:28.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.298584+0000 mon.a (mon.0) 1038 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:17:28.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.300396+0000 mon.a (mon.0) 1039 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:17:28.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.309861+0000 mon.a (mon.0) 1040 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:28.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.393213+0000 mon.a (mon.0) 1041 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:17:28.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.401861+0000 mon.a (mon.0) 1042 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:17:28.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.406888+0000 mon.a (mon.0) 1043 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.409754+0000 mon.a (mon.0) 1044 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.412098+0000 mon.a (mon.0) 1045 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.415982+0000 mon.a (mon.0) 1046 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.420347+0000 mon.a (mon.0) 1047 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.423538+0000 mon.a (mon.0) 1048 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.426444+0000 mon.a (mon.0) 1049 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.428940+0000 mon.a (mon.0) 1050 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.431417+0000 mon.a (mon.0) 1051 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.434175+0000 mon.a (mon.0) 1052 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.436568+0000 mon.a (mon.0) 1053 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.438994+0000 mon.a (mon.0) 1054 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:28 smithi192 bash[50362]: audit 2023-07-10T17:17:27.441188+0000 mon.a (mon.0) 1055 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.286035+0000 mon.a (mon.0) 1036 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.296698+0000 mon.a (mon.0) 1037 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:28.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.298584+0000 mon.a (mon.0) 1038 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:17:28.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.300396+0000 mon.a (mon.0) 1039 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:17:28.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.309861+0000 mon.a (mon.0) 1040 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:28.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.393213+0000 mon.a (mon.0) 1041 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:17:28.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.401861+0000 mon.a (mon.0) 1042 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:17:28.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.406888+0000 mon.a (mon.0) 1043 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.409754+0000 mon.a (mon.0) 1044 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.412098+0000 mon.a (mon.0) 1045 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.415982+0000 mon.a (mon.0) 1046 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.420347+0000 mon.a (mon.0) 1047 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.423538+0000 mon.a (mon.0) 1048 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.426444+0000 mon.a (mon.0) 1049 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.428940+0000 mon.a (mon.0) 1050 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.431417+0000 mon.a (mon.0) 1051 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.434175+0000 mon.a (mon.0) 1052 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.436568+0000 mon.a (mon.0) 1053 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.438994+0000 mon.a (mon.0) 1054 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[63900]: audit 2023-07-10T17:17:27.441188+0000 mon.a (mon.0) 1055 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.286035+0000 mon.a (mon.0) 1036 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:28.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.296698+0000 mon.a (mon.0) 1037 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:28.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.298584+0000 mon.a (mon.0) 1038 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:17:28.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.300396+0000 mon.a (mon.0) 1039 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:17:28.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.309861+0000 mon.a (mon.0) 1040 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:28.649 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.393213+0000 mon.a (mon.0) 1041 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:17:28.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.401861+0000 mon.a (mon.0) 1042 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:17:28.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.406888+0000 mon.a (mon.0) 1043 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.409754+0000 mon.a (mon.0) 1044 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.412098+0000 mon.a (mon.0) 1045 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.415982+0000 mon.a (mon.0) 1046 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.420347+0000 mon.a (mon.0) 1047 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.650 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.423538+0000 mon.a (mon.0) 1048 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.426444+0000 mon.a (mon.0) 1049 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.428940+0000 mon.a (mon.0) 1050 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.431417+0000 mon.a (mon.0) 1051 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.434175+0000 mon.a (mon.0) 1052 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.436568+0000 mon.a (mon.0) 1053 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.438994+0000 mon.a (mon.0) 1054 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:28.651 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:28 smithi116 bash[65980]: audit 2023-07-10T17:17:27.441188+0000 mon.a (mon.0) 1055 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:29.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:29 smithi192 bash[50362]: cephadm 2023-07-10T17:17:27.301871+0000 mgr.y (mgr.25390) 436 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:17:29.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:29 smithi192 bash[50362]: cluster 2023-07-10T17:17:27.303655+0000 mgr.y (mgr.25390) 437 : cluster 0 pgmap v320: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-10T17:17:29.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:29 smithi192 bash[50362]: audit 2023-07-10T17:17:27.393874+0000 mgr.y (mgr.25390) 438 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:17:29.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:29 smithi192 bash[50362]: cluster 2023-07-10T17:17:28.292206+0000 mon.a (mon.0) 1056 : cluster 1 Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2023-07-10T17:17:29.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:29 smithi192 bash[50362]: cluster 2023-07-10T17:17:28.292280+0000 mon.a (mon.0) 1057 : cluster 1 Cluster is now healthy 2023-07-10T17:17:29.644 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:29 smithi116 bash[63900]: cephadm 2023-07-10T17:17:27.301871+0000 mgr.y (mgr.25390) 436 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:17:29.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:29 smithi116 bash[63900]: cluster 2023-07-10T17:17:27.303655+0000 mgr.y (mgr.25390) 437 : cluster 0 pgmap v320: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-10T17:17:29.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:29 smithi116 bash[63900]: audit 2023-07-10T17:17:27.393874+0000 mgr.y (mgr.25390) 438 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:17:29.645 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:29 smithi116 bash[63900]: cluster 2023-07-10T17:17:28.292206+0000 mon.a (mon.0) 1056 : cluster 1 Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2023-07-10T17:17:29.646 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:29 smithi116 bash[63900]: cluster 2023-07-10T17:17:28.292280+0000 mon.a (mon.0) 1057 : cluster 1 Cluster is now healthy 2023-07-10T17:17:29.646 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:29 smithi116 bash[65980]: cephadm 2023-07-10T17:17:27.301871+0000 mgr.y (mgr.25390) 436 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:17:29.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:29 smithi116 bash[65980]: cluster 2023-07-10T17:17:27.303655+0000 mgr.y (mgr.25390) 437 : cluster 0 pgmap v320: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-10T17:17:29.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:29 smithi116 bash[65980]: audit 2023-07-10T17:17:27.393874+0000 mgr.y (mgr.25390) 438 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-10T17:17:29.647 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:29 smithi116 bash[65980]: cluster 2023-07-10T17:17:28.292206+0000 mon.a (mon.0) 1056 : cluster 1 Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2023-07-10T17:17:29.648 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:29 smithi116 bash[65980]: cluster 2023-07-10T17:17:28.292280+0000 mon.a (mon.0) 1057 : cluster 1 Cluster is now healthy 2023-07-10T17:17:30.598 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:17:31.354 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:17:31.355 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (10m) 14s ago 16m 11.0M - 0.20.0 0881eb8f169f 4c6ab4a7a2c3 2023-07-10T17:17:31.355 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (9m) 14s ago 16m 26.2M - 6.7.4 557c83e11646 9e506d6216ac 2023-07-10T17:17:31.355 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 running (72s) 14s ago 14m 47.7M - 3.6 17e8f0d79247 fbdecd8618b7 2023-07-10T17:17:31.356 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (8m) 14s ago 21m 431M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:17:31.356 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (9m) 14s ago 23m 485M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:17:31.356 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (8m) 14s ago 23m 83.8M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:17:31.357 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (6m) 14s ago 21m 63.6M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 74a082a2e795 2023-07-10T17:17:31.357 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (7m) 14s ago 21m 64.7M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:17:31.357 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (46s) 14s ago 17m 4759k - 1.5.0 0da6a335fe13 64b482421ffb 2023-07-10T17:17:31.358 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (42s) 14s ago 17m 4519k - 1.5.0 0da6a335fe13 82b166142066 2023-07-10T17:17:31.358 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (5m) 14s ago 20m 63.6M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 0562f587fbca 2023-07-10T17:17:31.358 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (5m) 14s ago 20m 63.2M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 5b2a12a0bfc9 2023-07-10T17:17:31.359 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (4m) 14s ago 20m 55.8M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 88096bc0faa9 2023-07-10T17:17:31.359 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (4m) 14s ago 19m 61.0M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 61ae280c471b 2023-07-10T17:17:31.359 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (3m) 14s ago 19m 61.7M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 44d809c18118 2023-07-10T17:17:31.359 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (3m) 14s ago 18m 57.5M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 179270eb7044 2023-07-10T17:17:31.360 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (2m) 14s ago 18m 55.0M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 b680bfbe9a0c 2023-07-10T17:17:31.360 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (2m) 14s ago 18m 57.0M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 bd3788803e68 2023-07-10T17:17:31.362 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 running (20s) 14s ago 17m 27.5M - 2.43.0 a07b618ecd1d 48c15c1430f7 2023-07-10T17:17:31.363 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (114s) 14s ago 14m 84.3M - 18.1.2-30-g8507faa7 17e8f0d79247 ef72a4e0ae29 2023-07-10T17:17:31.363 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (100s) 14s ago 14m 84.3M - 18.1.2-30-g8507faa7 17e8f0d79247 59b0f0b41f44 2023-07-10T17:17:31.363 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (107s) 14s ago 14m 84.2M - 18.1.2-30-g8507faa7 17e8f0d79247 42f4f263b75e 2023-07-10T17:17:31.363 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (94s) 14s ago 14m 83.9M - 18.1.2-30-g8507faa7 17e8f0d79247 2a257a88c4e9 2023-07-10T17:17:31.590 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:31 smithi116 bash[65980]: cluster 2023-07-10T17:17:29.304729+0000 mgr.y (mgr.25390) 439 : cluster 0 pgmap v321: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2023-07-10T17:17:31.591 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:31 smithi116 bash[65980]: cephadm 2023-07-10T17:17:29.913283+0000 mgr.y (mgr.25390) 440 : cephadm 1 Upgrade: Updating alertmanager.a 2023-07-10T17:17:31.591 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:31 smithi116 bash[65980]: cephadm 2023-07-10T17:17:29.931156+0000 mgr.y (mgr.25390) 441 : cephadm 1 Deploying daemon alertmanager.a on smithi116 2023-07-10T17:17:31.592 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:31 smithi116 bash[63900]: cluster 2023-07-10T17:17:29.304729+0000 mgr.y (mgr.25390) 439 : cluster 0 pgmap v321: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2023-07-10T17:17:31.592 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:31 smithi116 bash[63900]: cephadm 2023-07-10T17:17:29.913283+0000 mgr.y (mgr.25390) 440 : cephadm 1 Upgrade: Updating alertmanager.a 2023-07-10T17:17:31.592 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:31 smithi116 bash[63900]: cephadm 2023-07-10T17:17:29.931156+0000 mgr.y (mgr.25390) 441 : cephadm 1 Deploying daemon alertmanager.a on smithi116 2023-07-10T17:17:31.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:31 smithi192 bash[50362]: cluster 2023-07-10T17:17:29.304729+0000 mgr.y (mgr.25390) 439 : cluster 0 pgmap v321: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2023-07-10T17:17:31.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:31 smithi192 bash[50362]: cephadm 2023-07-10T17:17:29.913283+0000 mgr.y (mgr.25390) 440 : cephadm 1 Upgrade: Updating alertmanager.a 2023-07-10T17:17:31.642 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:31 smithi192 bash[50362]: cephadm 2023-07-10T17:17:29.931156+0000 mgr.y (mgr.25390) 441 : cephadm 1 Deploying daemon alertmanager.a on smithi116 2023-07-10T17:17:31.794 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:17:31.795 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:17:31.795 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:17:31.795 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:17:31.796 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:17:31.797 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:17:31.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:17:31.797 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:17:31.797 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 8 2023-07-10T17:17:31.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:17:31.797 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:17:31.797 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 4 2023-07-10T17:17:31.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:17:31.798 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:17:31.798 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 17 2023-07-10T17:17:31.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:17:31.798 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:17:32.164 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:17:32.165 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:17:32.165 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:17:32.165 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:17:32.165 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:17:32.165 INFO:teuthology.orchestra.run.smithi116.stdout: "iscsi", 2023-07-10T17:17:32.165 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw", 2023-07-10T17:17:32.165 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2023-07-10T17:17:32.166 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr", 2023-07-10T17:17:32.166 INFO:teuthology.orchestra.run.smithi116.stdout: "osd" 2023-07-10T17:17:32.166 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:17:32.166 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "18/23 daemons upgraded", 2023-07-10T17:17:32.166 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading alertmanager daemons", 2023-07-10T17:17:32.166 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:17:32.167 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:17:32.470 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:32 smithi116 bash[63900]: audit 2023-07-10T17:17:30.573221+0000 mgr.y (mgr.25390) 442 : audit 0 from='client.35127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:32.471 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:32 smithi116 bash[63900]: audit 2023-07-10T17:17:30.956974+0000 mgr.y (mgr.25390) 443 : audit 0 from='client.35133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:32.471 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:32 smithi116 bash[63900]: audit 2023-07-10T17:17:31.223629+0000 mgr.y (mgr.25390) 444 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:32.471 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:32 smithi116 bash[63900]: audit 2023-07-10T17:17:31.789332+0000 mon.c (mon.1) 35 : audit 0 from='client.45007 172.21.15.116:0/486719959' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:32.471 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:32 smithi116 bash[65980]: audit 2023-07-10T17:17:30.573221+0000 mgr.y (mgr.25390) 442 : audit 0 from='client.35127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:32.472 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:32 smithi116 bash[65980]: audit 2023-07-10T17:17:30.956974+0000 mgr.y (mgr.25390) 443 : audit 0 from='client.35133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:32.472 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:32 smithi116 bash[65980]: audit 2023-07-10T17:17:31.223629+0000 mgr.y (mgr.25390) 444 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:32.472 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:32 smithi116 bash[65980]: audit 2023-07-10T17:17:31.789332+0000 mon.c (mon.1) 35 : audit 0 from='client.45007 172.21.15.116:0/486719959' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:32.586 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2023-07-10T17:17:32.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:32 smithi192 bash[50362]: audit 2023-07-10T17:17:30.573221+0000 mgr.y (mgr.25390) 442 : audit 0 from='client.35127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:32.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:32 smithi192 bash[50362]: audit 2023-07-10T17:17:30.956974+0000 mgr.y (mgr.25390) 443 : audit 0 from='client.35133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:32.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:32 smithi192 bash[50362]: audit 2023-07-10T17:17:31.223629+0000 mgr.y (mgr.25390) 444 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:32.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:32 smithi192 bash[50362]: audit 2023-07-10T17:17:31.789332+0000 mon.c (mon.1) 35 : audit 0 from='client.45007 172.21.15.116:0/486719959' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:33.597 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:33 smithi116 bash[65980]: cluster 2023-07-10T17:17:31.305702+0000 mgr.y (mgr.25390) 445 : cluster 0 pgmap v322: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-10T17:17:33.598 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:33 smithi116 bash[65980]: audit 2023-07-10T17:17:31.324539+0000 mgr.y (mgr.25390) 446 : audit 0 from='client.35139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:33.598 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:33 smithi116 bash[65980]: audit 2023-07-10T17:17:32.158985+0000 mgr.y (mgr.25390) 447 : audit 0 from='client.45013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:33.598 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:33 smithi116 bash[65980]: audit 2023-07-10T17:17:32.581771+0000 mon.c (mon.1) 36 : audit 0 from='client.35157 172.21.15.116:0/3619252825' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:17:33.598 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:33 smithi116 bash[63900]: cluster 2023-07-10T17:17:31.305702+0000 mgr.y (mgr.25390) 445 : cluster 0 pgmap v322: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-10T17:17:33.598 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:33 smithi116 bash[63900]: audit 2023-07-10T17:17:31.324539+0000 mgr.y (mgr.25390) 446 : audit 0 from='client.35139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:33.599 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:33 smithi116 bash[63900]: audit 2023-07-10T17:17:32.158985+0000 mgr.y (mgr.25390) 447 : audit 0 from='client.45013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:33.599 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:33 smithi116 bash[63900]: audit 2023-07-10T17:17:32.581771+0000 mon.c (mon.1) 36 : audit 0 from='client.35157 172.21.15.116:0/3619252825' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:17:33.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:33 smithi192 bash[50362]: cluster 2023-07-10T17:17:31.305702+0000 mgr.y (mgr.25390) 445 : cluster 0 pgmap v322: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-10T17:17:33.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:33 smithi192 bash[50362]: audit 2023-07-10T17:17:31.324539+0000 mgr.y (mgr.25390) 446 : audit 0 from='client.35139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:33.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:33 smithi192 bash[50362]: audit 2023-07-10T17:17:32.158985+0000 mgr.y (mgr.25390) 447 : audit 0 from='client.45013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:17:33.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:33 smithi192 bash[50362]: audit 2023-07-10T17:17:32.581771+0000 mon.c (mon.1) 36 : audit 0 from='client.35157 172.21.15.116:0/3619252825' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:17:34.545 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:34 smithi116 bash[65980]: cluster 2023-07-10T17:17:33.306424+0000 mgr.y (mgr.25390) 448 : cluster 0 pgmap v323: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2023-07-10T17:17:34.546 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:34 smithi116 bash[63900]: cluster 2023-07-10T17:17:33.306424+0000 mgr.y (mgr.25390) 448 : cluster 0 pgmap v323: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2023-07-10T17:17:34.546 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:17:34 smithi116 systemd[1]: Stopping Ceph alertmanager.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:17:34.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:34 smithi192 bash[50362]: cluster 2023-07-10T17:17:33.306424+0000 mgr.y (mgr.25390) 448 : cluster 0 pgmap v323: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2023-07-10T17:17:34.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:17:34 smithi116 bash[57058]: level=info ts=2023-07-10T17:17:34.540Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2023-07-10T17:17:35.594 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:17:35 smithi116 bash[88157]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-alertmanager.a 2023-07-10T17:17:35.594 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:17:35 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@alertmanager.a.service: Succeeded. 2023-07-10T17:17:35.594 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:17:35 smithi116 systemd[1]: Stopped Ceph alertmanager.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:17:35.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:17:35 smithi116 systemd[1]: Started Ceph alertmanager.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:17:36.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:36 smithi192 bash[50362]: cluster 2023-07-10T17:17:35.307573+0000 mgr.y (mgr.25390) 449 : cluster 0 pgmap v324: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2023-07-10T17:17:36.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:36 smithi192 bash[50362]: audit 2023-07-10T17:17:35.629735+0000 mon.a (mon.0) 1058 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:36.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:36 smithi192 bash[50362]: audit 2023-07-10T17:17:35.637316+0000 mon.a (mon.0) 1059 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:36.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:36 smithi116 bash[63900]: cluster 2023-07-10T17:17:35.307573+0000 mgr.y (mgr.25390) 449 : cluster 0 pgmap v324: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2023-07-10T17:17:36.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:36 smithi116 bash[63900]: audit 2023-07-10T17:17:35.629735+0000 mon.a (mon.0) 1058 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:36.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:36 smithi116 bash[63900]: audit 2023-07-10T17:17:35.637316+0000 mon.a (mon.0) 1059 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:36.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:36 smithi116 bash[65980]: cluster 2023-07-10T17:17:35.307573+0000 mgr.y (mgr.25390) 449 : cluster 0 pgmap v324: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2023-07-10T17:17:36.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:36 smithi116 bash[65980]: audit 2023-07-10T17:17:35.629735+0000 mon.a (mon.0) 1058 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:36.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:36 smithi116 bash[65980]: audit 2023-07-10T17:17:35.637316+0000 mon.a (mon.0) 1059 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:37.394 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:17:37 smithi116 bash[88313]: ts=2023-07-10T17:17:37.044Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2023-07-10T17:17:37.395 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:17:37 smithi116 bash[88313]: ts=2023-07-10T17:17:37.044Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2023-07-10T17:17:37.395 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:17:37 smithi116 bash[88313]: ts=2023-07-10T17:17:37.046Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.116 port=9094 2023-07-10T17:17:37.395 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:17:37 smithi116 bash[88313]: ts=2023-07-10T17:17:37.049Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2023-07-10T17:17:37.395 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:17:37 smithi116 bash[88313]: ts=2023-07-10T17:17:37.097Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-10T17:17:37.395 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:17:37 smithi116 bash[88313]: ts=2023-07-10T17:17:37.098Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-10T17:17:37.395 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:17:37 smithi116 bash[88313]: ts=2023-07-10T17:17:37.105Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2023-07-10T17:17:37.395 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:17:37 smithi116 bash[88313]: ts=2023-07-10T17:17:37.105Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2023-07-10T17:17:39.054 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:38 smithi116 bash[63900]: cluster 2023-07-10T17:17:37.309090+0000 mgr.y (mgr.25390) 450 : cluster 0 pgmap v325: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-10T17:17:39.055 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:38 smithi116 bash[65980]: cluster 2023-07-10T17:17:37.309090+0000 mgr.y (mgr.25390) 450 : cluster 0 pgmap v325: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-10T17:17:39.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:38 smithi192 bash[50362]: cluster 2023-07-10T17:17:37.309090+0000 mgr.y (mgr.25390) 450 : cluster 0 pgmap v325: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-10T17:17:39.394 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:17:39 smithi116 bash[88313]: ts=2023-07-10T17:17:39.050Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000898454s 2023-07-10T17:17:41.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:40 smithi192 bash[50362]: cluster 2023-07-10T17:17:39.310066+0000 mgr.y (mgr.25390) 451 : cluster 0 pgmap v326: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:41.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:40 smithi116 bash[63900]: cluster 2023-07-10T17:17:39.310066+0000 mgr.y (mgr.25390) 451 : cluster 0 pgmap v326: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:41.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:40 smithi116 bash[65980]: cluster 2023-07-10T17:17:39.310066+0000 mgr.y (mgr.25390) 451 : cluster 0 pgmap v326: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:42.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:41 smithi192 bash[50362]: audit 2023-07-10T17:17:41.231740+0000 mgr.y (mgr.25390) 452 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:42.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:41 smithi116 bash[65980]: audit 2023-07-10T17:17:41.231740+0000 mgr.y (mgr.25390) 452 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:42.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:41 smithi116 bash[63900]: audit 2023-07-10T17:17:41.231740+0000 mgr.y (mgr.25390) 452 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:43.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:42 smithi192 bash[50362]: cluster 2023-07-10T17:17:41.311440+0000 mgr.y (mgr.25390) 453 : cluster 0 pgmap v327: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:43.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:42 smithi116 bash[63900]: cluster 2023-07-10T17:17:41.311440+0000 mgr.y (mgr.25390) 453 : cluster 0 pgmap v327: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:43.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:42 smithi116 bash[65980]: cluster 2023-07-10T17:17:41.311440+0000 mgr.y (mgr.25390) 453 : cluster 0 pgmap v327: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:44.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:44 smithi192 bash[50362]: audit 2023-07-10T17:17:43.018269+0000 mon.a (mon.0) 1060 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:44.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:44 smithi192 bash[50362]: audit 2023-07-10T17:17:43.029374+0000 mon.a (mon.0) 1061 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:44.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:44 smithi192 bash[50362]: audit 2023-07-10T17:17:43.830122+0000 mon.a (mon.0) 1062 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:44.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:44 smithi192 bash[50362]: audit 2023-07-10T17:17:43.842507+0000 mon.a (mon.0) 1063 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:44.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:44 smithi116 bash[63900]: audit 2023-07-10T17:17:43.018269+0000 mon.a (mon.0) 1060 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:44.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:44 smithi116 bash[63900]: audit 2023-07-10T17:17:43.029374+0000 mon.a (mon.0) 1061 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:44.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:44 smithi116 bash[63900]: audit 2023-07-10T17:17:43.830122+0000 mon.a (mon.0) 1062 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:44.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:44 smithi116 bash[63900]: audit 2023-07-10T17:17:43.842507+0000 mon.a (mon.0) 1063 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:44.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:44 smithi116 bash[65980]: audit 2023-07-10T17:17:43.018269+0000 mon.a (mon.0) 1060 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:44.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:44 smithi116 bash[65980]: audit 2023-07-10T17:17:43.029374+0000 mon.a (mon.0) 1061 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:44.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:44 smithi116 bash[65980]: audit 2023-07-10T17:17:43.830122+0000 mon.a (mon.0) 1062 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:44.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:44 smithi116 bash[65980]: audit 2023-07-10T17:17:43.842507+0000 mon.a (mon.0) 1063 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:45.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:45 smithi192 bash[50362]: cluster 2023-07-10T17:17:43.312636+0000 mgr.y (mgr.25390) 454 : cluster 0 pgmap v328: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:45.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:45 smithi116 bash[63900]: cluster 2023-07-10T17:17:43.312636+0000 mgr.y (mgr.25390) 454 : cluster 0 pgmap v328: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:45.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:45 smithi116 bash[65980]: cluster 2023-07-10T17:17:43.312636+0000 mgr.y (mgr.25390) 454 : cluster 0 pgmap v328: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:47.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:47 smithi192 bash[50362]: cluster 2023-07-10T17:17:45.313881+0000 mgr.y (mgr.25390) 455 : cluster 0 pgmap v329: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:47.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:47 smithi116 bash[63900]: cluster 2023-07-10T17:17:45.313881+0000 mgr.y (mgr.25390) 455 : cluster 0 pgmap v329: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:47.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:47 smithi116 bash[65980]: cluster 2023-07-10T17:17:45.313881+0000 mgr.y (mgr.25390) 455 : cluster 0 pgmap v329: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:47.396 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:17:47 smithi116 bash[88313]: ts=2023-07-10T17:17:47.052Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003249074s 2023-07-10T17:17:49.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:49 smithi192 bash[50362]: cluster 2023-07-10T17:17:47.315395+0000 mgr.y (mgr.25390) 456 : cluster 0 pgmap v330: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:49.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:49 smithi116 bash[63900]: cluster 2023-07-10T17:17:47.315395+0000 mgr.y (mgr.25390) 456 : cluster 0 pgmap v330: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:49.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:49 smithi116 bash[65980]: cluster 2023-07-10T17:17:47.315395+0000 mgr.y (mgr.25390) 456 : cluster 0 pgmap v330: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:51.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:51 smithi116 bash[63900]: cluster 2023-07-10T17:17:49.316325+0000 mgr.y (mgr.25390) 457 : cluster 0 pgmap v331: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:51.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:51 smithi116 bash[65980]: cluster 2023-07-10T17:17:49.316325+0000 mgr.y (mgr.25390) 457 : cluster 0 pgmap v331: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:51.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:51 smithi192 bash[50362]: cluster 2023-07-10T17:17:49.316325+0000 mgr.y (mgr.25390) 457 : cluster 0 pgmap v331: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:52.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:52 smithi116 bash[63900]: audit 2023-07-10T17:17:51.238124+0000 mgr.y (mgr.25390) 458 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:52.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:52 smithi116 bash[65980]: audit 2023-07-10T17:17:51.238124+0000 mgr.y (mgr.25390) 458 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:52.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:52 smithi192 bash[50362]: audit 2023-07-10T17:17:51.238124+0000 mgr.y (mgr.25390) 458 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:17:53.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:53 smithi116 bash[65980]: cluster 2023-07-10T17:17:51.317601+0000 mgr.y (mgr.25390) 459 : cluster 0 pgmap v332: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:53.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:53 smithi116 bash[63900]: cluster 2023-07-10T17:17:51.317601+0000 mgr.y (mgr.25390) 459 : cluster 0 pgmap v332: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:53.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:53 smithi192 bash[50362]: cluster 2023-07-10T17:17:51.317601+0000 mgr.y (mgr.25390) 459 : cluster 0 pgmap v332: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:54.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: cluster 2023-07-10T17:17:53.318809+0000 mgr.y (mgr.25390) 460 : cluster 0 pgmap v333: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:54.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.493446+0000 mon.a (mon.0) 1064 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:54.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.505797+0000 mon.a (mon.0) 1065 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:54.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.507797+0000 mon.a (mon.0) 1066 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:17:54.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.509381+0000 mon.a (mon.0) 1067 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:17:54.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: cephadm 2023-07-10T17:17:53.510841+0000 mgr.y (mgr.25390) 461 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:17:54.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.520107+0000 mon.a (mon.0) 1068 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:54.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.604308+0000 mon.a (mon.0) 1069 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:17:54.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.605252+0000 mgr.y (mgr.25390) 462 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:17:54.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.618733+0000 mon.a (mon.0) 1070 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:17:54.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.624020+0000 mon.a (mon.0) 1071 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.626767+0000 mon.a (mon.0) 1072 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.628862+0000 mon.a (mon.0) 1073 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.632533+0000 mon.a (mon.0) 1074 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.636628+0000 mon.a (mon.0) 1075 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.892 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.639604+0000 mon.a (mon.0) 1076 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.893 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.641883+0000 mon.a (mon.0) 1077 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.893 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.644012+0000 mon.a (mon.0) 1078 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.894 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.646152+0000 mon.a (mon.0) 1079 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.894 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.648510+0000 mon.a (mon.0) 1080 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.894 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.650908+0000 mon.a (mon.0) 1081 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.895 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.654166+0000 mon.a (mon.0) 1082 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.895 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.656964+0000 mon.a (mon.0) 1083 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.895 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:54 smithi192 bash[50362]: audit 2023-07-10T17:17:53.659622+0000 mon.a (mon.0) 1084 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: cluster 2023-07-10T17:17:53.318809+0000 mgr.y (mgr.25390) 460 : cluster 0 pgmap v333: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:54.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.493446+0000 mon.a (mon.0) 1064 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:54.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.505797+0000 mon.a (mon.0) 1065 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:54.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.507797+0000 mon.a (mon.0) 1066 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:17:54.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.509381+0000 mon.a (mon.0) 1067 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:17:54.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: cephadm 2023-07-10T17:17:53.510841+0000 mgr.y (mgr.25390) 461 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:17:54.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.520107+0000 mon.a (mon.0) 1068 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:54.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.604308+0000 mon.a (mon.0) 1069 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:17:54.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.605252+0000 mgr.y (mgr.25390) 462 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:17:54.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.618733+0000 mon.a (mon.0) 1070 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:17:54.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.624020+0000 mon.a (mon.0) 1071 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.626767+0000 mon.a (mon.0) 1072 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.628862+0000 mon.a (mon.0) 1073 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.897 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.632533+0000 mon.a (mon.0) 1074 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.636628+0000 mon.a (mon.0) 1075 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.639604+0000 mon.a (mon.0) 1076 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.641883+0000 mon.a (mon.0) 1077 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.644012+0000 mon.a (mon.0) 1078 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.646152+0000 mon.a (mon.0) 1079 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.898 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: cluster 2023-07-10T17:17:53.318809+0000 mgr.y (mgr.25390) 460 : cluster 0 pgmap v333: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:54.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.493446+0000 mon.a (mon.0) 1064 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:54.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.505797+0000 mon.a (mon.0) 1065 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:54.899 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.507797+0000 mon.a (mon.0) 1066 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:17:54.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.509381+0000 mon.a (mon.0) 1067 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:17:54.900 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: cephadm 2023-07-10T17:17:53.510841+0000 mgr.y (mgr.25390) 461 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:17:54.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.520107+0000 mon.a (mon.0) 1068 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:17:54.901 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.604308+0000 mon.a (mon.0) 1069 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:17:54.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.605252+0000 mgr.y (mgr.25390) 462 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-10T17:17:54.902 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.618733+0000 mon.a (mon.0) 1070 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:17:54.904 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.624020+0000 mon.a (mon.0) 1071 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.904 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.626767+0000 mon.a (mon.0) 1072 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.904 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.628862+0000 mon.a (mon.0) 1073 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.905 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.632533+0000 mon.a (mon.0) 1074 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.905 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.636628+0000 mon.a (mon.0) 1075 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.905 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.639604+0000 mon.a (mon.0) 1076 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.905 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.641883+0000 mon.a (mon.0) 1077 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.905 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.644012+0000 mon.a (mon.0) 1078 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.905 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.646152+0000 mon.a (mon.0) 1079 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.648510+0000 mon.a (mon.0) 1080 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.650908+0000 mon.a (mon.0) 1081 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.654166+0000 mon.a (mon.0) 1082 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.656964+0000 mon.a (mon.0) 1083 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[63900]: audit 2023-07-10T17:17:53.659622+0000 mon.a (mon.0) 1084 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.648510+0000 mon.a (mon.0) 1080 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.650908+0000 mon.a (mon.0) 1081 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.654166+0000 mon.a (mon.0) 1082 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.656964+0000 mon.a (mon.0) 1083 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:54.907 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:54 smithi116 bash[65980]: audit 2023-07-10T17:17:53.659622+0000 mon.a (mon.0) 1084 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:17:56.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:56 smithi192 bash[50362]: cluster 2023-07-10T17:17:55.319960+0000 mgr.y (mgr.25390) 463 : cluster 0 pgmap v334: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:56.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:56 smithi192 bash[50362]: cephadm 2023-07-10T17:17:56.032046+0000 mgr.y (mgr.25390) 464 : cephadm 1 Upgrade: Updating grafana.a 2023-07-10T17:17:56.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:56 smithi192 bash[50362]: cephadm 2023-07-10T17:17:56.056766+0000 mgr.y (mgr.25390) 465 : cephadm 1 Deploying daemon grafana.a on smithi192 2023-07-10T17:17:56.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:56 smithi116 bash[65980]: cluster 2023-07-10T17:17:55.319960+0000 mgr.y (mgr.25390) 463 : cluster 0 pgmap v334: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:56.894 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:56 smithi116 bash[65980]: cephadm 2023-07-10T17:17:56.032046+0000 mgr.y (mgr.25390) 464 : cephadm 1 Upgrade: Updating grafana.a 2023-07-10T17:17:56.895 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:56 smithi116 bash[65980]: cephadm 2023-07-10T17:17:56.056766+0000 mgr.y (mgr.25390) 465 : cephadm 1 Deploying daemon grafana.a on smithi192 2023-07-10T17:17:56.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:56 smithi116 bash[63900]: cluster 2023-07-10T17:17:55.319960+0000 mgr.y (mgr.25390) 463 : cluster 0 pgmap v334: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:17:56.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:56 smithi116 bash[63900]: cephadm 2023-07-10T17:17:56.032046+0000 mgr.y (mgr.25390) 464 : cephadm 1 Upgrade: Updating grafana.a 2023-07-10T17:17:56.896 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:56 smithi116 bash[63900]: cephadm 2023-07-10T17:17:56.056766+0000 mgr.y (mgr.25390) 465 : cephadm 1 Deploying daemon grafana.a on smithi192 2023-07-10T17:17:59.053 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:17:58 smithi192 bash[50362]: cluster 2023-07-10T17:17:57.321464+0000 mgr.y (mgr.25390) 466 : cluster 0 pgmap v335: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:59.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:17:58 smithi116 bash[63900]: cluster 2023-07-10T17:17:57.321464+0000 mgr.y (mgr.25390) 466 : cluster 0 pgmap v335: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:17:59.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:17:58 smithi116 bash[65980]: cluster 2023-07-10T17:17:57.321464+0000 mgr.y (mgr.25390) 466 : cluster 0 pgmap v335: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:00.314 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:00 smithi192 systemd[1]: Stopping Ceph grafana.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:18:00.644 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:00 smithi192 bash[45347]: t=2023-07-10T17:18:00+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2023-07-10T17:18:00.645 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:00 smithi192 bash[45347]: t=2023-07-10T17:18:00+0000 lvl=info msg="Stopped NotificationService" logger=server reason="context canceled" 2023-07-10T17:18:00.645 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:00 smithi192 bash[45347]: t=2023-07-10T17:18:00+0000 lvl=info msg="Stopped CleanUpService" logger=server reason="context canceled" 2023-07-10T17:18:00.645 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:00 smithi192 bash[45347]: t=2023-07-10T17:18:00+0000 lvl=info msg="Stopped InternalMetricsService" logger=server reason="context canceled" 2023-07-10T17:18:00.645 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:00 smithi192 bash[45347]: t=2023-07-10T17:18:00+0000 lvl=info msg="Stopped PluginManager" logger=server reason="context canceled" 2023-07-10T17:18:00.645 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:00 smithi192 bash[45347]: t=2023-07-10T17:18:00+0000 lvl=info msg="Stopped BackendPluginManager" logger=server reason="context canceled" 2023-07-10T17:18:00.645 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:00 smithi192 bash[45347]: t=2023-07-10T17:18:00+0000 lvl=info msg="Stopped provisioningServiceImpl" logger=server reason="context canceled" 2023-07-10T17:18:00.646 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:00 smithi192 bash[45347]: t=2023-07-10T17:18:00+0000 lvl=info msg="Stopped RemoteCache" logger=server reason="context canceled" 2023-07-10T17:18:00.646 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:00 smithi192 bash[45347]: t=2023-07-10T17:18:00+0000 lvl=info msg="Stopped UsageStatsService" logger=server reason="context canceled" 2023-07-10T17:18:00.646 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:00 smithi192 bash[45347]: t=2023-07-10T17:18:00+0000 lvl=info msg="Stopped AlertEngine" logger=server reason="context canceled" 2023-07-10T17:18:00.646 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:00 smithi192 bash[45347]: t=2023-07-10T17:18:00+0000 lvl=info msg="Stopped Stream Manager" 2023-07-10T17:18:00.646 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:00 smithi192 bash[45347]: t=2023-07-10T17:18:00+0000 lvl=info msg="Stopped UserAuthTokenService" logger=server reason="context canceled" 2023-07-10T17:18:00.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:00 smithi192 bash[50362]: cluster 2023-07-10T17:17:59.322348+0000 mgr.y (mgr.25390) 467 : cluster 0 pgmap v336: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:01.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:00 smithi116 bash[63900]: cluster 2023-07-10T17:17:59.322348+0000 mgr.y (mgr.25390) 467 : cluster 0 pgmap v336: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:01.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:00 smithi116 bash[65980]: cluster 2023-07-10T17:17:59.322348+0000 mgr.y (mgr.25390) 467 : cluster 0 pgmap v336: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:01.338 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:00 smithi192 bash[68103]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-grafana.a 2023-07-10T17:18:01.338 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:01 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@grafana.a.service: Succeeded. 2023-07-10T17:18:01.338 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:01 smithi192 systemd[1]: Stopped Ceph grafana.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:18:01.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:01 smithi192 systemd[1]: Started Ceph grafana.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:18:02.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:01 smithi192 bash[50362]: audit 2023-07-10T17:18:01.249420+0000 mgr.y (mgr.25390) 468 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:02.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:01 smithi192 bash[50362]: audit 2023-07-10T17:18:01.373095+0000 mon.a (mon.0) 1085 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:02.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:01 smithi192 bash[50362]: audit 2023-07-10T17:18:01.382242+0000 mon.a (mon.0) 1086 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:02.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:01 smithi116 bash[65980]: audit 2023-07-10T17:18:01.249420+0000 mgr.y (mgr.25390) 468 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:02.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:01 smithi116 bash[65980]: audit 2023-07-10T17:18:01.373095+0000 mon.a (mon.0) 1085 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:02.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:01 smithi116 bash[65980]: audit 2023-07-10T17:18:01.382242+0000 mon.a (mon.0) 1086 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:02.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:01 smithi116 bash[63900]: audit 2023-07-10T17:18:01.249420+0000 mgr.y (mgr.25390) 468 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:02.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:01 smithi116 bash[63900]: audit 2023-07-10T17:18:01.373095+0000 mon.a (mon.0) 1085 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:02.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:01 smithi116 bash[63900]: audit 2023-07-10T17:18:01.382242+0000 mon.a (mon.0) 1086 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:02.811 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[50362]: cluster 2023-07-10T17:18:01.323648+0000 mgr.y (mgr.25390) 469 : cluster 0 pgmap v337: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:02.983 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:18:03.075 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=settings t=2023-07-10T17:18:02.807836594Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2023-07-10T17:18:03.076 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=settings t=2023-07-10T17:18:02.808512266Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2023-07-10T17:18:03.083 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=settings t=2023-07-10T17:18:02.808585618Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2023-07-10T17:18:03.084 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=settings t=2023-07-10T17:18:02.808629144Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2023-07-10T17:18:03.084 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=settings t=2023-07-10T17:18:02.808664627Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2023-07-10T17:18:03.084 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=settings t=2023-07-10T17:18:02.80870706Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2023-07-10T17:18:03.085 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=settings t=2023-07-10T17:18:02.808743016Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2023-07-10T17:18:03.085 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=settings t=2023-07-10T17:18:02.808783645Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2023-07-10T17:18:03.085 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=settings t=2023-07-10T17:18:02.808825061Z level=info msg="Path Home" path=/usr/share/grafana 2023-07-10T17:18:03.086 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=settings t=2023-07-10T17:18:02.808859764Z level=info msg="Path Data" path=/var/lib/grafana 2023-07-10T17:18:03.086 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=settings t=2023-07-10T17:18:02.808897393Z level=info msg="Path Logs" path=/var/log/grafana 2023-07-10T17:18:03.086 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=settings t=2023-07-10T17:18:02.808929149Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2023-07-10T17:18:03.087 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=settings t=2023-07-10T17:18:02.808964396Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2023-07-10T17:18:03.087 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=settings t=2023-07-10T17:18:02.809000358Z level=info msg="App mode production" 2023-07-10T17:18:03.087 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=sqlstore t=2023-07-10T17:18:02.809172095Z level=info msg="Connecting to DB" dbtype=sqlite3 2023-07-10T17:18:03.088 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=sqlstore t=2023-07-10T17:18:02.809260644Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2023-07-10T17:18:03.088 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:02.811534149Z level=info msg="Starting DB migrations" 2023-07-10T17:18:03.088 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:02.811767185Z level=info msg="Executing migration" id="create migration_log table" 2023-07-10T17:18:03.088 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:02.84412383Z level=info msg="Executing migration" id="create user table" 2023-07-10T17:18:03.089 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:02.869391027Z level=info msg="Executing migration" id="add unique index user.login" 2023-07-10T17:18:03.089 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:02.894509734Z level=info msg="Executing migration" id="add unique index user.email" 2023-07-10T17:18:03.089 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:02.919868425Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2023-07-10T17:18:03.089 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:02.945076214Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2023-07-10T17:18:03.090 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:02.970218969Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2023-07-10T17:18:03.090 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:02 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:02.995625317Z level=info msg="Executing migration" id="create user table v2" 2023-07-10T17:18:03.090 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.020834164Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2023-07-10T17:18:03.091 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.046118467Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2023-07-10T17:18:03.091 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.071384964Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2023-07-10T17:18:03.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:02 smithi116 bash[63900]: cluster 2023-07-10T17:18:01.323648+0000 mgr.y (mgr.25390) 469 : cluster 0 pgmap v337: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:03.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:02 smithi116 bash[65980]: cluster 2023-07-10T17:18:01.323648+0000 mgr.y (mgr.25390) 469 : cluster 0 pgmap v337: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:03.334 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.113256643Z level=info msg="Executing migration" id="Drop old table user_v1" 2023-07-10T17:18:03.335 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.138685827Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2023-07-10T17:18:03.335 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.180606598Z level=info msg="Executing migration" id="Update user table charset" 2023-07-10T17:18:03.335 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.205880351Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2023-07-10T17:18:03.335 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.289455781Z level=info msg="Executing migration" id="Add missing user data" 2023-07-10T17:18:03.335 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.331620735Z level=info msg="Executing migration" id="Add is_disabled column to user" 2023-07-10T17:18:03.612 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.356881885Z level=info msg="Executing migration" id="Add index user.login/user.email" 2023-07-10T17:18:03.613 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.398883317Z level=info msg="Executing migration" id="Add is_service_account column to user" 2023-07-10T17:18:03.613 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.42417911Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2023-07-10T17:18:03.613 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.4578362Z level=info msg="Executing migration" id="create temp user table v1-7" 2023-07-10T17:18:03.613 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.499870593Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2023-07-10T17:18:03.613 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.524964417Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2023-07-10T17:18:03.613 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.567106366Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2023-07-10T17:18:03.614 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.609031873Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2023-07-10T17:18:03.725 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:18:03.726 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (28s) 20s ago 17m 15.6M - 0.25.0 c8568f914cd2 5032fe494a7b 2023-07-10T17:18:03.726 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 starting - - - - 2023-07-10T17:18:03.726 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 running (104s) 20s ago 15m 48.0M - 3.6 17e8f0d79247 fbdecd8618b7 2023-07-10T17:18:03.726 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (8m) 46s ago 21m 431M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:18:03.726 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (9m) 20s ago 23m 487M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:18:03.726 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (8m) 20s ago 23m 84.1M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:18:03.726 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (7m) 46s ago 22m 63.6M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 74a082a2e795 2023-07-10T17:18:03.727 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (7m) 20s ago 22m 63.0M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:18:03.727 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (78s) 20s ago 17m 9140k - 1.5.0 0da6a335fe13 64b482421ffb 2023-07-10T17:18:03.727 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (75s) 46s ago 17m 4519k - 1.5.0 0da6a335fe13 82b166142066 2023-07-10T17:18:03.727 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (6m) 20s ago 21m 63.7M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 0562f587fbca 2023-07-10T17:18:03.727 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (5m) 20s ago 21m 63.5M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 5b2a12a0bfc9 2023-07-10T17:18:03.727 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (5m) 20s ago 20m 56.1M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 88096bc0faa9 2023-07-10T17:18:03.727 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (4m) 20s ago 20m 61.2M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 61ae280c471b 2023-07-10T17:18:03.727 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (4m) 46s ago 19m 61.7M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 44d809c18118 2023-07-10T17:18:03.728 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (3m) 46s ago 19m 57.5M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 179270eb7044 2023-07-10T17:18:03.728 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (3m) 46s ago 18m 55.0M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 b680bfbe9a0c 2023-07-10T17:18:03.728 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (2m) 46s ago 18m 57.0M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 bd3788803e68 2023-07-10T17:18:03.728 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 running (53s) 46s ago 17m 27.5M - 2.43.0 a07b618ecd1d 48c15c1430f7 2023-07-10T17:18:03.728 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (2m) 20s ago 15m 84.3M - 18.1.2-30-g8507faa7 17e8f0d79247 ef72a4e0ae29 2023-07-10T17:18:03.728 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (2m) 46s ago 15m 84.3M - 18.1.2-30-g8507faa7 17e8f0d79247 59b0f0b41f44 2023-07-10T17:18:03.728 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (2m) 20s ago 15m 84.4M - 18.1.2-30-g8507faa7 17e8f0d79247 42f4f263b75e 2023-07-10T17:18:03.729 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (2m) 46s ago 15m 83.9M - 18.1.2-30-g8507faa7 17e8f0d79247 2a257a88c4e9 2023-07-10T17:18:03.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[50362]: audit 2023-07-10T17:18:02.962545+0000 mgr.y (mgr.25390) 470 : audit 0 from='client.35163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:03.886 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.634359069Z level=info msg="Executing migration" id="Update temp_user table charset" 2023-07-10T17:18:03.886 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.676381565Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2023-07-10T17:18:03.886 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.743558775Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2023-07-10T17:18:03.886 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.768773593Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2023-07-10T17:18:03.886 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.802242903Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2023-07-10T17:18:03.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.827495327Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2023-07-10T17:18:03.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.875344575Z level=info msg="Executing migration" id="create temp_user v2" 2023-07-10T17:18:04.133 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.90310209Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2023-07-10T17:18:04.133 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.928455902Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2023-07-10T17:18:04.134 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.962220919Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2023-07-10T17:18:04.134 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:03 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:03.987560768Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2023-07-10T17:18:04.134 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.021141716Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2023-07-10T17:18:04.134 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.046238551Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2023-07-10T17:18:04.134 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.0882666Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2023-07-10T17:18:04.134 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.130256069Z level=info msg="Executing migration" id="create star table" 2023-07-10T17:18:04.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:03 smithi116 bash[65980]: audit 2023-07-10T17:18:02.962545+0000 mgr.y (mgr.25390) 470 : audit 0 from='client.35163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:04.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:03 smithi116 bash[63900]: audit 2023-07-10T17:18:02.962545+0000 mgr.y (mgr.25390) 470 : audit 0 from='client.35163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:04.164 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:18:04.164 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:18:04.164 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:18:04.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:18:04.165 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:18:04.165 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:18:04.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:18:04.165 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:18:04.165 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 8 2023-07-10T17:18:04.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:18:04.166 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:18:04.166 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 4 2023-07-10T17:18:04.166 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:18:04.166 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:18:04.166 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 17 2023-07-10T17:18:04.166 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:18:04.166 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:18:04.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.155577088Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2023-07-10T17:18:04.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.231183584Z level=info msg="Executing migration" id="create org table v1" 2023-07-10T17:18:04.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.25625458Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2023-07-10T17:18:04.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.281574012Z level=info msg="Executing migration" id="create org_user table v1" 2023-07-10T17:18:04.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.323531499Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2023-07-10T17:18:04.388 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.370246441Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2023-07-10T17:18:04.541 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:18:04.541 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8507faa7ce284e46218d481e3ee9c411ac3dc32c", 2023-07-10T17:18:04.541 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": true, 2023-07-10T17:18:04.541 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "Upgrading all daemon types on all hosts", 2023-07-10T17:18:04.541 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [ 2023-07-10T17:18:04.542 INFO:teuthology.orchestra.run.smithi116.stdout: "iscsi", 2023-07-10T17:18:04.542 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw", 2023-07-10T17:18:04.542 INFO:teuthology.orchestra.run.smithi116.stdout: "mon", 2023-07-10T17:18:04.542 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr", 2023-07-10T17:18:04.542 INFO:teuthology.orchestra.run.smithi116.stdout: "osd" 2023-07-10T17:18:04.542 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2023-07-10T17:18:04.542 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": "18/23 daemons upgraded", 2023-07-10T17:18:04.542 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "Currently upgrading grafana daemons", 2023-07-10T17:18:04.543 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:18:04.543 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:18:04.703 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.420576825Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2023-07-10T17:18:04.703 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.462830499Z level=info msg="Executing migration" id="Update org table charset" 2023-07-10T17:18:04.703 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.488110449Z level=info msg="Executing migration" id="Update org_user table charset" 2023-07-10T17:18:04.703 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.530124784Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2023-07-10T17:18:04.703 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.580494253Z level=info msg="Executing migration" id="create dashboard table" 2023-07-10T17:18:04.704 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.605800279Z level=info msg="Executing migration" id="add index dashboard.account_id" 2023-07-10T17:18:04.704 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.631025212Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2023-07-10T17:18:04.962 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[50362]: cluster 2023-07-10T17:18:03.324388+0000 mgr.y (mgr.25390) 471 : cluster 0 pgmap v338: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:04.962 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[50362]: audit 2023-07-10T17:18:03.339496+0000 mgr.y (mgr.25390) 472 : audit 0 from='client.35169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:04.962 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[50362]: audit 2023-07-10T17:18:03.702577+0000 mgr.y (mgr.25390) 473 : audit 0 from='client.45037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:04.963 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[50362]: audit 2023-07-10T17:18:04.159332+0000 mon.c (mon.1) 37 : audit 0 from='client.45043 172.21.15.116:0/3004104437' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:04.963 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.714617827Z level=info msg="Executing migration" id="create dashboard_tag table" 2023-07-10T17:18:04.963 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.739894994Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2023-07-10T17:18:04.963 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.782070975Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2023-07-10T17:18:04.963 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.80726255Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2023-07-10T17:18:04.963 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.832554473Z level=info msg="Executing migration" id="create dashboard v2" 2023-07-10T17:18:04.964 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.857810916Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2023-07-10T17:18:04.964 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.883022537Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2023-07-10T17:18:04.964 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.908263343Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2023-07-10T17:18:04.964 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.933476154Z level=info msg="Executing migration" id="drop table dashboard_v1" 2023-07-10T17:18:04.964 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.958817485Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2023-07-10T17:18:04.981 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2023-07-10T17:18:05.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:04 smithi116 bash[63900]: cluster 2023-07-10T17:18:03.324388+0000 mgr.y (mgr.25390) 471 : cluster 0 pgmap v338: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:05.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:04 smithi116 bash[63900]: audit 2023-07-10T17:18:03.339496+0000 mgr.y (mgr.25390) 472 : audit 0 from='client.35169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:05.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:04 smithi116 bash[63900]: audit 2023-07-10T17:18:03.702577+0000 mgr.y (mgr.25390) 473 : audit 0 from='client.45037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:05.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:04 smithi116 bash[63900]: audit 2023-07-10T17:18:04.159332+0000 mon.c (mon.1) 37 : audit 0 from='client.45043 172.21.15.116:0/3004104437' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:05.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:04 smithi116 bash[65980]: cluster 2023-07-10T17:18:03.324388+0000 mgr.y (mgr.25390) 471 : cluster 0 pgmap v338: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:05.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:04 smithi116 bash[65980]: audit 2023-07-10T17:18:03.339496+0000 mgr.y (mgr.25390) 472 : audit 0 from='client.35169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:05.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:04 smithi116 bash[65980]: audit 2023-07-10T17:18:03.702577+0000 mgr.y (mgr.25390) 473 : audit 0 from='client.45037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:05.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:04 smithi116 bash[65980]: audit 2023-07-10T17:18:04.159332+0000 mon.c (mon.1) 37 : audit 0 from='client.45043 172.21.15.116:0/3004104437' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:05.230 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:04 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:04.984015546Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2023-07-10T17:18:05.231 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.026011035Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2023-07-10T17:18:05.231 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.051308187Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2023-07-10T17:18:05.231 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.076682851Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2023-07-10T17:18:05.231 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.118513506Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2023-07-10T17:18:05.231 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.202129745Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2023-07-10T17:18:05.231 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.227416671Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2023-07-10T17:18:05.506 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.269379089Z level=info msg="Executing migration" id="Update dashboard table charset" 2023-07-10T17:18:05.507 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.294695184Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2023-07-10T17:18:05.507 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.319902005Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2023-07-10T17:18:05.507 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.34521228Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2023-07-10T17:18:05.508 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.392605001Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2023-07-10T17:18:05.508 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.417861031Z level=info msg="Executing migration" id="Add column uid in dashboard" 2023-07-10T17:18:05.508 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.459767968Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2023-07-10T17:18:05.509 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.501863374Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2023-07-10T17:18:05.757 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.52714283Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2023-07-10T17:18:05.757 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.552343957Z level=info msg="Executing migration" id="Update dashboard title length" 2023-07-10T17:18:05.757 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.594337522Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2023-07-10T17:18:05.757 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.669656355Z level=info msg="Executing migration" id="create dashboard_provisioning" 2023-07-10T17:18:05.758 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.711392226Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2023-07-10T17:18:05.758 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.753873777Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2023-07-10T17:18:05.758 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[50362]: audit 2023-07-10T17:18:04.535521+0000 mgr.y (mgr.25390) 474 : audit 0 from='client.35187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:05.758 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[50362]: audit 2023-07-10T17:18:04.976373+0000 mon.a (mon.0) 1087 : audit 0 from='client.45055 172.21.15.116:0/1514437379' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:18:06.009 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.77889089Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2023-07-10T17:18:06.010 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.804202158Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2023-07-10T17:18:06.010 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.829479478Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2023-07-10T17:18:06.011 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.85497324Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2023-07-10T17:18:06.011 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.88025688Z level=info msg="Executing migration" id="Add check_sum column" 2023-07-10T17:18:06.011 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.905415531Z level=info msg="Executing migration" id="Add index for dashboard_title" 2023-07-10T17:18:06.011 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.930568072Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2023-07-10T17:18:06.012 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.955959619Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2023-07-10T17:18:06.012 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:05 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:05.981177856Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2023-07-10T17:18:06.012 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.006289851Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2023-07-10T17:18:06.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:05 smithi116 bash[63900]: audit 2023-07-10T17:18:04.535521+0000 mgr.y (mgr.25390) 474 : audit 0 from='client.35187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:06.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:05 smithi116 bash[63900]: audit 2023-07-10T17:18:04.976373+0000 mon.a (mon.0) 1087 : audit 0 from='client.45055 172.21.15.116:0/1514437379' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:18:06.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:05 smithi116 bash[65980]: audit 2023-07-10T17:18:04.535521+0000 mgr.y (mgr.25390) 474 : audit 0 from='client.35187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:06.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:05 smithi116 bash[65980]: audit 2023-07-10T17:18:04.976373+0000 mon.a (mon.0) 1087 : audit 0 from='client.45055 172.21.15.116:0/1514437379' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:18:06.278 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.031706053Z level=info msg="Executing migration" id="create data_source table" 2023-07-10T17:18:06.280 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.065321872Z level=info msg="Executing migration" id="add index data_source.account_id" 2023-07-10T17:18:06.280 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.140628693Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2023-07-10T17:18:06.281 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.182553449Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2023-07-10T17:18:06.281 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.207759695Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2023-07-10T17:18:06.281 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.233072377Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2023-07-10T17:18:06.281 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.275050509Z level=info msg="Executing migration" id="create data_source table v2" 2023-07-10T17:18:06.553 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.31701383Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2023-07-10T17:18:06.554 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.380938935Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2023-07-10T17:18:06.554 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.406215671Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2023-07-10T17:18:06.554 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.431580702Z level=info msg="Executing migration" id="Add column with_credentials" 2023-07-10T17:18:06.554 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.473577006Z level=info msg="Executing migration" id="Add secure json data column" 2023-07-10T17:18:06.555 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.498831655Z level=info msg="Executing migration" id="Update data_source table charset" 2023-07-10T17:18:06.555 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.524144126Z level=info msg="Executing migration" id="Update initial version to 1" 2023-07-10T17:18:06.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[50362]: cluster 2023-07-10T17:18:05.325341+0000 mgr.y (mgr.25390) 475 : cluster 0 pgmap v339: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:06.821 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.549387903Z level=info msg="Executing migration" id="Add read_only data column" 2023-07-10T17:18:06.821 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.624632221Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2023-07-10T17:18:06.822 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.649963282Z level=info msg="Executing migration" id="Update json_data with nulls" 2023-07-10T17:18:06.822 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.691865081Z level=info msg="Executing migration" id="Add uid column" 2023-07-10T17:18:06.822 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.725521241Z level=info msg="Executing migration" id="Update uid value" 2023-07-10T17:18:06.822 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.751015004Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2023-07-10T17:18:06.822 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.776178233Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2023-07-10T17:18:06.822 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.818121428Z level=info msg="Executing migration" id="create api_key table" 2023-07-10T17:18:07.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.84342152Z level=info msg="Executing migration" id="add index api_key.account_id" 2023-07-10T17:18:07.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.868620943Z level=info msg="Executing migration" id="add index api_key.key" 2023-07-10T17:18:07.108 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.910813092Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2023-07-10T17:18:07.108 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.952885076Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2023-07-10T17:18:07.108 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:06 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:06.978027048Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2023-07-10T17:18:07.109 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.003327585Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2023-07-10T17:18:07.109 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.028689046Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2023-07-10T17:18:07.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:06 smithi116 bash[63900]: cluster 2023-07-10T17:18:05.325341+0000 mgr.y (mgr.25390) 475 : cluster 0 pgmap v339: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:07.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:06 smithi116 bash[65980]: cluster 2023-07-10T17:18:05.325341+0000 mgr.y (mgr.25390) 475 : cluster 0 pgmap v339: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:07.369 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.104024413Z level=info msg="Executing migration" id="create api_key table v2" 2023-07-10T17:18:07.370 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.14595921Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2023-07-10T17:18:07.370 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.188193913Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2023-07-10T17:18:07.370 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.213491241Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2023-07-10T17:18:07.370 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.264005419Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2023-07-10T17:18:07.370 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.313402602Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2023-07-10T17:18:07.370 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.339160565Z level=info msg="Executing migration" id="Update api_key table charset" 2023-07-10T17:18:07.371 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.364478429Z level=info msg="Executing migration" id="Add expires to api_key table" 2023-07-10T17:18:07.640 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.406360528Z level=info msg="Executing migration" id="Add service account foreign key" 2023-07-10T17:18:07.641 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.431660952Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2023-07-10T17:18:07.641 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.4569454Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2023-07-10T17:18:07.641 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.482103177Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2023-07-10T17:18:07.641 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.523992518Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2023-07-10T17:18:07.641 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.599401035Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2023-07-10T17:18:07.919 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.641498951Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2023-07-10T17:18:07.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.664027801Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2023-07-10T17:18:07.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.714646167Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2023-07-10T17:18:07.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.756674757Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2023-07-10T17:18:07.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.781910289Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2023-07-10T17:18:07.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.823834504Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2023-07-10T17:18:07.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.849250523Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2023-07-10T17:18:07.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.874284882Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2023-07-10T17:18:07.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.916279386Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2023-07-10T17:18:08.196 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.941572337Z level=info msg="Executing migration" id="create quota table v1" 2023-07-10T17:18:08.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:07 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:07.983400001Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2023-07-10T17:18:08.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.008635714Z level=info msg="Executing migration" id="Update quota table charset" 2023-07-10T17:18:08.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.084023035Z level=info msg="Executing migration" id="create plugin_setting table" 2023-07-10T17:18:08.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.126105855Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2023-07-10T17:18:08.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.151295471Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2023-07-10T17:18:08.197 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.193255762Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2023-07-10T17:18:08.455 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.218665464Z level=info msg="Executing migration" id="create session table" 2023-07-10T17:18:08.456 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.260667983Z level=info msg="Executing migration" id="Drop old table playlist table" 2023-07-10T17:18:08.456 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.300756872Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2023-07-10T17:18:08.456 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.342874303Z level=info msg="Executing migration" id="create playlist table v2" 2023-07-10T17:18:08.456 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.384990604Z level=info msg="Executing migration" id="create playlist item table v2" 2023-07-10T17:18:08.456 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.410136831Z level=info msg="Executing migration" id="Update playlist table charset" 2023-07-10T17:18:08.716 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[50362]: cluster 2023-07-10T17:18:07.326971+0000 mgr.y (mgr.25390) 476 : cluster 0 pgmap v340: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:08.716 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[50362]: audit 2023-07-10T17:18:07.685533+0000 mon.a (mon.0) 1088 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:08.717 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[50362]: audit 2023-07-10T17:18:07.694754+0000 mon.a (mon.0) 1089 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:08.717 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[50362]: audit 2023-07-10T17:18:08.496713+0000 mon.a (mon.0) 1090 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:08.717 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[50362]: audit 2023-07-10T17:18:08.509933+0000 mon.a (mon.0) 1091 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:08.717 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.4519784Z level=info msg="Executing migration" id="Update playlist_item table charset" 2023-07-10T17:18:08.717 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.477398668Z level=info msg="Executing migration" id="drop preferences table v2" 2023-07-10T17:18:08.717 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.552616676Z level=info msg="Executing migration" id="drop preferences table v3" 2023-07-10T17:18:08.718 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.578047635Z level=info msg="Executing migration" id="create preferences table v3" 2023-07-10T17:18:08.720 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.619948912Z level=info msg="Executing migration" id="Update preferences table charset" 2023-07-10T17:18:08.721 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.645189211Z level=info msg="Executing migration" id="Add column team_id in preferences" 2023-07-10T17:18:08.721 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.67039672Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2023-07-10T17:18:08.721 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.712543957Z level=info msg="Executing migration" id="Add column week_start in preferences" 2023-07-10T17:18:09.034 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.737817229Z level=info msg="Executing migration" id="Add column preferences.json_data" 2023-07-10T17:18:09.035 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.779649618Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2023-07-10T17:18:09.035 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.82858796Z level=info msg="Executing migration" id="Add preferences index org_id" 2023-07-10T17:18:09.037 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.853861379Z level=info msg="Executing migration" id="Add preferences index user_id" 2023-07-10T17:18:09.037 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.879166739Z level=info msg="Executing migration" id="create alert table v1" 2023-07-10T17:18:09.037 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.904393995Z level=info msg="Executing migration" id="add index alert org_id & id " 2023-07-10T17:18:09.037 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.929542141Z level=info msg="Executing migration" id="add index alert state" 2023-07-10T17:18:09.037 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:08 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:08.954791913Z level=info msg="Executing migration" id="add index alert dashboard_id" 2023-07-10T17:18:09.038 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.030128682Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2023-07-10T17:18:09.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:08 smithi116 bash[63900]: cluster 2023-07-10T17:18:07.326971+0000 mgr.y (mgr.25390) 476 : cluster 0 pgmap v340: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:09.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:08 smithi116 bash[63900]: audit 2023-07-10T17:18:07.685533+0000 mon.a (mon.0) 1088 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:09.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:08 smithi116 bash[63900]: audit 2023-07-10T17:18:07.694754+0000 mon.a (mon.0) 1089 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:09.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:08 smithi116 bash[63900]: audit 2023-07-10T17:18:08.496713+0000 mon.a (mon.0) 1090 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:09.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:08 smithi116 bash[63900]: audit 2023-07-10T17:18:08.509933+0000 mon.a (mon.0) 1091 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:09.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:08 smithi116 bash[65980]: cluster 2023-07-10T17:18:07.326971+0000 mgr.y (mgr.25390) 476 : cluster 0 pgmap v340: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:09.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:08 smithi116 bash[65980]: audit 2023-07-10T17:18:07.685533+0000 mon.a (mon.0) 1088 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:09.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:08 smithi116 bash[65980]: audit 2023-07-10T17:18:07.694754+0000 mon.a (mon.0) 1089 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:09.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:08 smithi116 bash[65980]: audit 2023-07-10T17:18:08.496713+0000 mon.a (mon.0) 1090 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:09.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:08 smithi116 bash[65980]: audit 2023-07-10T17:18:08.509933+0000 mon.a (mon.0) 1091 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:09.301 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.055358684Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2023-07-10T17:18:09.301 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.097238528Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2023-07-10T17:18:09.301 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.139173348Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2023-07-10T17:18:09.301 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.164471379Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2023-07-10T17:18:09.302 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.206767366Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2023-07-10T17:18:09.302 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.232094883Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2023-07-10T17:18:09.302 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.257184753Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2023-07-10T17:18:09.303 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.297841268Z level=info msg="Executing migration" id="create alert_notification table v1" 2023-07-10T17:18:09.569 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.33970755Z level=info msg="Executing migration" id="Add column is_default" 2023-07-10T17:18:09.570 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.365204573Z level=info msg="Executing migration" id="Add column frequency" 2023-07-10T17:18:09.570 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.390406835Z level=info msg="Executing migration" id="Add column send_reminder" 2023-07-10T17:18:09.570 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.415777231Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2023-07-10T17:18:09.570 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.44099834Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2023-07-10T17:18:09.570 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.524614525Z level=info msg="Executing migration" id="Update alert table charset" 2023-07-10T17:18:09.570 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.566441454Z level=info msg="Executing migration" id="Update alert_notification table charset" 2023-07-10T17:18:09.820 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.608492532Z level=info msg="Executing migration" id="create notification_journal table v1" 2023-07-10T17:18:09.821 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.650660452Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2023-07-10T17:18:09.821 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.675859187Z level=info msg="Executing migration" id="drop alert_notification_journal" 2023-07-10T17:18:09.821 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.717820413Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2023-07-10T17:18:09.821 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.743030536Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2023-07-10T17:18:09.821 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.79219129Z level=info msg="Executing migration" id="Add for to alert table" 2023-07-10T17:18:10.098 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.817405421Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2023-07-10T17:18:10.098 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.859590522Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2023-07-10T17:18:10.098 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.884727061Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2023-07-10T17:18:10.098 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.910015165Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2023-07-10T17:18:10.098 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:09 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:09.976896055Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2023-07-10T17:18:10.099 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.002211392Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2023-07-10T17:18:10.099 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.027423617Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2023-07-10T17:18:10.099 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.052676115Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2023-07-10T17:18:10.099 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.094584331Z level=info msg="Executing migration" id="Drop old annotation table v4" 2023-07-10T17:18:10.374 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.136585576Z level=info msg="Executing migration" id="create annotation table v5" 2023-07-10T17:18:10.375 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.178644313Z level=info msg="Executing migration" id="add index annotation 0 v3" 2023-07-10T17:18:10.375 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.203784912Z level=info msg="Executing migration" id="add index annotation 1 v3" 2023-07-10T17:18:10.376 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.229096282Z level=info msg="Executing migration" id="add index annotation 2 v3" 2023-07-10T17:18:10.377 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.251636892Z level=info msg="Executing migration" id="add index annotation 3 v3" 2023-07-10T17:18:10.377 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.293674106Z level=info msg="Executing migration" id="add index annotation 4 v3" 2023-07-10T17:18:10.377 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.318959757Z level=info msg="Executing migration" id="Update annotation table charset" 2023-07-10T17:18:10.377 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.344080746Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2023-07-10T17:18:10.378 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.369279838Z level=info msg="Executing migration" id="Drop category_id index" 2023-07-10T17:18:10.624 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.394537905Z level=info msg="Executing migration" id="Add column tags to annotation table" 2023-07-10T17:18:10.624 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.478173575Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2023-07-10T17:18:10.624 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.520223549Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2023-07-10T17:18:10.625 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.545465726Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2023-07-10T17:18:10.625 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.570762097Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2023-07-10T17:18:10.625 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.620997331Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2023-07-10T17:18:10.877 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[50362]: cluster 2023-07-10T17:18:09.327931+0000 mgr.y (mgr.25390) 477 : cluster 0 pgmap v341: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:10.877 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.662974389Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2023-07-10T17:18:10.878 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.705080974Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2023-07-10T17:18:10.878 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.730210432Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2023-07-10T17:18:10.879 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.755543302Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2023-07-10T17:18:10.879 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.780762117Z level=info msg="Executing migration" id="Add created time to annotation table" 2023-07-10T17:18:10.879 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.822791004Z level=info msg="Executing migration" id="Add updated time to annotation table" 2023-07-10T17:18:10.879 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.848038455Z level=info msg="Executing migration" id="Add index for created in annotation table" 2023-07-10T17:18:10.879 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.873254863Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2023-07-10T17:18:11.135 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.946302931Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2023-07-10T17:18:11.135 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:10 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:10.988429777Z level=info msg="Executing migration" id="Add epoch_end column" 2023-07-10T17:18:11.136 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.013643033Z level=info msg="Executing migration" id="Add index for epoch_end" 2023-07-10T17:18:11.136 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.038927759Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2023-07-10T17:18:11.136 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.089139788Z level=info msg="Executing migration" id="Move region to single row" 2023-07-10T17:18:11.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:10 smithi116 bash[63900]: cluster 2023-07-10T17:18:09.327931+0000 mgr.y (mgr.25390) 477 : cluster 0 pgmap v341: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:11.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:10 smithi116 bash[65980]: cluster 2023-07-10T17:18:09.327931+0000 mgr.y (mgr.25390) 477 : cluster 0 pgmap v341: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:11.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.13109047Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2023-07-10T17:18:11.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.173105234Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2023-07-10T17:18:11.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.198351114Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2023-07-10T17:18:11.388 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.223682567Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2023-07-10T17:18:11.388 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.274291879Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2023-07-10T17:18:11.388 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.299412916Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2023-07-10T17:18:11.388 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.324702698Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2023-07-10T17:18:11.388 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.349971715Z level=info msg="Executing migration" id="create test_data table" 2023-07-10T17:18:11.695 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.425346991Z level=info msg="Executing migration" id="create dashboard_version table v1" 2023-07-10T17:18:11.696 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.450468536Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2023-07-10T17:18:11.696 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.475741309Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2023-07-10T17:18:11.696 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.51775387Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2023-07-10T17:18:11.696 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.557866673Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2023-07-10T17:18:11.696 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.599746153Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2023-07-10T17:18:11.696 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.624924794Z level=info msg="Executing migration" id="create team table" 2023-07-10T17:18:11.696 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.667133655Z level=info msg="Executing migration" id="add index team.org_id" 2023-07-10T17:18:11.697 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.692306344Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2023-07-10T17:18:11.964 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[50362]: audit 2023-07-10T17:18:11.258713+0000 mgr.y (mgr.25390) 478 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:11.965 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.734286978Z level=info msg="Executing migration" id="create team member table" 2023-07-10T17:18:11.965 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.759648471Z level=info msg="Executing migration" id="add index team_member.org_id" 2023-07-10T17:18:11.965 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.784791276Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2023-07-10T17:18:11.966 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.810124072Z level=info msg="Executing migration" id="add index team_member.team_id" 2023-07-10T17:18:11.966 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.835387269Z level=info msg="Executing migration" id="Add column email to team table" 2023-07-10T17:18:11.966 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.910593772Z level=info msg="Executing migration" id="Add column external to team_member table" 2023-07-10T17:18:11.966 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.935755188Z level=info msg="Executing migration" id="Add column permission to team_member table" 2023-07-10T17:18:12.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:11 smithi116 bash[63900]: audit 2023-07-10T17:18:11.258713+0000 mgr.y (mgr.25390) 478 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:12.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:11 smithi116 bash[65980]: audit 2023-07-10T17:18:11.258713+0000 mgr.y (mgr.25390) 478 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:12.223 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:11 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:11.960938049Z level=info msg="Executing migration" id="create dashboard acl table" 2023-07-10T17:18:12.223 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.003052679Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2023-07-10T17:18:12.223 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.045036256Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2023-07-10T17:18:12.223 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.068515406Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2023-07-10T17:18:12.223 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.11047139Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2023-07-10T17:18:12.223 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.152593042Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2023-07-10T17:18:12.224 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.177816462Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2023-07-10T17:18:12.224 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.219693275Z level=info msg="Executing migration" id="add index dashboard_permission" 2023-07-10T17:18:12.476 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.261783023Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2023-07-10T17:18:12.477 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.303710092Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2023-07-10T17:18:12.477 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.32889143Z level=info msg="Executing migration" id="create tag table" 2023-07-10T17:18:12.477 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.42096822Z level=info msg="Executing migration" id="add index tag.key_value" 2023-07-10T17:18:12.477 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.446255443Z level=info msg="Executing migration" id="create login attempt table" 2023-07-10T17:18:12.477 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.471443598Z level=info msg="Executing migration" id="add index login_attempt.username" 2023-07-10T17:18:12.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[50362]: cluster 2023-07-10T17:18:11.329259+0000 mgr.y (mgr.25390) 479 : cluster 0 pgmap v342: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:12.742 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.496680021Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2023-07-10T17:18:12.742 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.538603813Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2023-07-10T17:18:12.742 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.572290054Z level=info msg="Executing migration" id="create login_attempt v2" 2023-07-10T17:18:12.742 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.614172943Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2023-07-10T17:18:12.743 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.654617013Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2023-07-10T17:18:12.743 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.696576511Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2023-07-10T17:18:12.743 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.738705655Z level=info msg="Executing migration" id="create user auth table" 2023-07-10T17:18:13.006 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.780647544Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2023-07-10T17:18:13.006 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.847963399Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2023-07-10T17:18:13.006 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.88129639Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2023-07-10T17:18:13.006 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.9066473Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2023-07-10T17:18:13.007 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.931870055Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2023-07-10T17:18:13.007 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.957113126Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2023-07-10T17:18:13.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:12 smithi116 bash[63900]: cluster 2023-07-10T17:18:11.329259+0000 mgr.y (mgr.25390) 479 : cluster 0 pgmap v342: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:13.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:12 smithi116 bash[65980]: cluster 2023-07-10T17:18:11.329259+0000 mgr.y (mgr.25390) 479 : cluster 0 pgmap v342: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:13.277 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:12 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:12.999176762Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2023-07-10T17:18:13.277 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:13 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:13.032685692Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2023-07-10T17:18:13.278 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:13 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:13.057947258Z level=info msg="Executing migration" id="create server_lock table" 2023-07-10T17:18:13.278 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:13 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:13.100159148Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2023-07-10T17:18:13.278 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:13 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:13.139524522Z level=info msg="Executing migration" id="create user auth token table" 2023-07-10T17:18:13.278 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:13 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:13.164832917Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2023-07-10T17:18:13.278 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:13 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:13.190046904Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2023-07-10T17:18:13.278 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:13 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:13.231942869Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2023-07-10T17:18:13.589 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:13 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:13.274093046Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2023-07-10T17:18:13.589 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:13 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:13.349406813Z level=info msg="Executing migration" id="create cache_data table" 2023-07-10T17:18:13.589 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:13 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:13.374694252Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2023-07-10T17:18:13.590 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:13 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:13.416958084Z level=info msg="Executing migration" id="create short_url table v1" 2023-07-10T17:18:13.590 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:13 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:13.459000493Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2023-07-10T17:18:13.741 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:13 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:13.584940266Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2023-07-10T17:18:13.742 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:13 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:13.638166853Z level=info msg="Executing migration" id="delete alert_definition table" 2023-07-10T17:18:13.742 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:13 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:13.73819047Z level=info msg="Executing migration" id="recreate alert_definition table" 2023-07-10T17:18:14.263 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:13 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:13.890203031Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2023-07-10T17:18:14.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:14 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:14.259939816Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2023-07-10T17:18:14.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:14 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:14.427611498Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2023-07-10T17:18:14.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:14 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:14.49484214Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2023-07-10T17:18:14.982 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:14 smithi192 bash[50362]: cluster 2023-07-10T17:18:13.330545+0000 mgr.y (mgr.25390) 480 : cluster 0 pgmap v343: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:14.983 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:14 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:14.796080596Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2023-07-10T17:18:14.983 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:14 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:14.896816168Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2023-07-10T17:18:15.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:14 smithi116 bash[63900]: cluster 2023-07-10T17:18:13.330545+0000 mgr.y (mgr.25390) 480 : cluster 0 pgmap v343: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:15.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:14 smithi116 bash[65980]: cluster 2023-07-10T17:18:13.330545+0000 mgr.y (mgr.25390) 480 : cluster 0 pgmap v343: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:15.284 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:14 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:14.979147071Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2023-07-10T17:18:15.285 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:15 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:15.054670662Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2023-07-10T17:18:15.285 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:15 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:15.142512224Z level=info msg="Executing migration" id="drop alert_definition table" 2023-07-10T17:18:15.285 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:15 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:15.201857362Z level=info msg="Executing migration" id="delete alert_definition_version table" 2023-07-10T17:18:15.542 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:15 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:15.280964305Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2023-07-10T17:18:15.543 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:15 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:15.372894385Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2023-07-10T17:18:15.543 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:15 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:15.398175826Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2023-07-10T17:18:15.543 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:15 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:15.440415269Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2023-07-10T17:18:15.543 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:15 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:15.482515021Z level=info msg="Executing migration" id="drop alert_definition_version table" 2023-07-10T17:18:15.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:15 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:15.548630898Z level=info msg="Executing migration" id="create alert_instance table" 2023-07-10T17:18:15.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:15 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:15.65002229Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2023-07-10T17:18:15.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:15 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:15.758528121Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2023-07-10T17:18:16.137 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:15 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:15.958437615Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2023-07-10T17:18:16.539 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:16 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:16.234717431Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2023-07-10T17:18:16.539 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:16 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:16.4203772Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2023-07-10T17:18:16.539 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:16 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:16.467232251Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2023-07-10T17:18:16.540 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:16 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:16.534844092Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2023-07-10T17:18:16.809 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:16 smithi192 bash[50362]: cluster 2023-07-10T17:18:15.331729+0000 mgr.y (mgr.25390) 481 : cluster 0 pgmap v344: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:16.809 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:16 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:16.602774323Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2023-07-10T17:18:16.809 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:16 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:16.627865827Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2023-07-10T17:18:16.809 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:16 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:16.670127025Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2023-07-10T17:18:16.810 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:16 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:16.805804978Z level=info msg="Executing migration" id="create alert_rule table" 2023-07-10T17:18:17.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:16 smithi116 bash[63900]: cluster 2023-07-10T17:18:15.331729+0000 mgr.y (mgr.25390) 481 : cluster 0 pgmap v344: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:17.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:16 smithi116 bash[65980]: cluster 2023-07-10T17:18:15.331729+0000 mgr.y (mgr.25390) 481 : cluster 0 pgmap v344: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:17.386 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:17 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:17.125656701Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2023-07-10T17:18:17.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:17 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:17.292606788Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2023-07-10T17:18:17.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:17 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:17.36891322Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2023-07-10T17:18:17.987 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:17 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:17.657238382Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2023-07-10T17:18:17.987 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:17 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:17.773734567Z level=info msg="Executing migration" id="add column for to alert_rule" 2023-07-10T17:18:17.987 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:17 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:17.885343383Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2023-07-10T17:18:17.988 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:17 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:17.983866737Z level=info msg="Executing migration" id="add column labels to alert_rule" 2023-07-10T17:18:18.289 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.110192246Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2023-07-10T17:18:18.290 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.196259841Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2023-07-10T17:18:18.591 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.285906068Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2023-07-10T17:18:18.591 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.36287358Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2023-07-10T17:18:18.591 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.389729424Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2023-07-10T17:18:18.591 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.420831401Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2023-07-10T17:18:18.591 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.481804789Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2023-07-10T17:18:18.592 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.587558813Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2023-07-10T17:18:18.877 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[50362]: cluster 2023-07-10T17:18:17.333292+0000 mgr.y (mgr.25390) 482 : cluster 0 pgmap v345: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:18.877 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.620799306Z level=info msg="Executing migration" id="create alert_rule_version table" 2023-07-10T17:18:18.877 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.661440129Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2023-07-10T17:18:18.877 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.703799694Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2023-07-10T17:18:18.878 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.75559712Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2023-07-10T17:18:18.878 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.797467022Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2023-07-10T17:18:18.878 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.826272482Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2023-07-10T17:18:18.878 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.873374739Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2023-07-10T17:18:19.137 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.923799766Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2023-07-10T17:18:19.137 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.953661362Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2023-07-10T17:18:19.137 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:18 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:18.985645892Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2023-07-10T17:18:19.137 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:19 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:19.060569916Z level=info msg="Executing migration" id=create_alert_configuration_table 2023-07-10T17:18:19.137 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:19 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:19.106030749Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2023-07-10T17:18:19.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:18 smithi116 bash[63900]: cluster 2023-07-10T17:18:17.333292+0000 mgr.y (mgr.25390) 482 : cluster 0 pgmap v345: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:19.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:18 smithi116 bash[65980]: cluster 2023-07-10T17:18:17.333292+0000 mgr.y (mgr.25390) 482 : cluster 0 pgmap v345: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:19.470 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:19 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:19.146110304Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2023-07-10T17:18:19.471 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:19 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:19.188155198Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2023-07-10T17:18:19.471 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:19 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:19.230419217Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2023-07-10T17:18:19.471 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:19 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:19.272354403Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2023-07-10T17:18:19.471 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:19 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:19.367043841Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2023-07-10T17:18:19.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:19 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:19.467035171Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2023-07-10T17:18:19.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:19 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:19.663922392Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2023-07-10T17:18:20.297 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:19 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:19.994686269Z level=info msg="Executing migration" id="create provenance_type table" 2023-07-10T17:18:20.297 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.169563096Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2023-07-10T17:18:20.297 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.226617792Z level=info msg="Executing migration" id="create alert_image table" 2023-07-10T17:18:20.558 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.293927222Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2023-07-10T17:18:20.559 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.363635612Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2023-07-10T17:18:20.559 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.405649905Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2023-07-10T17:18:20.559 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.453474121Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2023-07-10T17:18:20.559 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.479401253Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2023-07-10T17:18:20.559 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.47976765Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2023-07-10T17:18:20.560 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.554740224Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2023-07-10T17:18:20.819 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.580030988Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2023-07-10T17:18:20.819 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.62185211Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2023-07-10T17:18:20.819 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.622106612Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2023-07-10T17:18:20.819 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.622395036Z level=info msg="alerts found to migrate" alerts=0 2023-07-10T17:18:20.820 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.663908606Z level=info msg="Executing migration" id="create library_element table v1" 2023-07-10T17:18:20.820 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.706106805Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2023-07-10T17:18:20.820 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.748000592Z level=info msg="Executing migration" id="create library_element_connection table v1" 2023-07-10T17:18:20.820 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.790134928Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2023-07-10T17:18:20.820 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.815414745Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2023-07-10T17:18:20.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: cluster 2023-07-10T17:18:19.334274+0000 mgr.y (mgr.25390) 483 : cluster 0 pgmap v346: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:20.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.416762+0000 mon.a (mon.0) 1092 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:20.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.430317+0000 mon.a (mon.0) 1093 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:20.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.432060+0000 mon.a (mon.0) 1094 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:18:20.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.433919+0000 mon.a (mon.0) 1095 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:18:20.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.445597+0000 mon.a (mon.0) 1096 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:20.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.529776+0000 mon.a (mon.0) 1097 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:18:20.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.538808+0000 mon.a (mon.0) 1098 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:18:20.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.543970+0000 mon.a (mon.0) 1099 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.546957+0000 mon.a (mon.0) 1100 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.549244+0000 mon.a (mon.0) 1101 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.552920+0000 mon.a (mon.0) 1102 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.557172+0000 mon.a (mon.0) 1103 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.560196+0000 mon.a (mon.0) 1104 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.562316+0000 mon.a (mon.0) 1105 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.564347+0000 mon.a (mon.0) 1106 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.567345+0000 mon.a (mon.0) 1107 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.570675+0000 mon.a (mon.0) 1108 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.573491+0000 mon.a (mon.0) 1109 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.825 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.577173+0000 mon.a (mon.0) 1110 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.825 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.580397+0000 mon.a (mon.0) 1111 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.825 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.583472+0000 mon.a (mon.0) 1112 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.825 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.586600+0000 mon.a (mon.0) 1113 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.825 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.589450+0000 mon.a (mon.0) 1114 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.825 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.592204+0000 mon.a (mon.0) 1115 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:20.825 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.605786+0000 mon.a (mon.0) 1116 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:20.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.607590+0000 mon.a (mon.0) 1117 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2023-07-10T17:18:20.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.615397+0000 mon.a (mon.0) 1118 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2023-07-10T17:18:20.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.616904+0000 mon.a (mon.0) 1119 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:20.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.626225+0000 mon.a (mon.0) 1120 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2023-07-10T17:18:20.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.627605+0000 mon.a (mon.0) 1121 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2023-07-10T17:18:20.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.635443+0000 mon.a (mon.0) 1122 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2023-07-10T17:18:20.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.637201+0000 mon.a (mon.0) 1123 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2023-07-10T17:18:20.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.645982+0000 mon.a (mon.0) 1124 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2023-07-10T17:18:20.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.647758+0000 mon.a (mon.0) 1125 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2023-07-10T17:18:20.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.656156+0000 mon.a (mon.0) 1126 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2023-07-10T17:18:20.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.657484+0000 mon.a (mon.0) 1127 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2023-07-10T17:18:20.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.665948+0000 mon.a (mon.0) 1128 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2023-07-10T17:18:20.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.667495+0000 mon.a (mon.0) 1129 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2023-07-10T17:18:20.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.675711+0000 mon.a (mon.0) 1130 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2023-07-10T17:18:20.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.676910+0000 mon.a (mon.0) 1131 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:20.828 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.678007+0000 mon.a (mon.0) 1132 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2023-07-10T17:18:20.828 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.685818+0000 mon.a (mon.0) 1133 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2023-07-10T17:18:20.828 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.687101+0000 mon.a (mon.0) 1134 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2023-07-10T17:18:20.828 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.695800+0000 mon.a (mon.0) 1135 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2023-07-10T17:18:20.829 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.697425+0000 mon.a (mon.0) 1136 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2023-07-10T17:18:20.829 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.706355+0000 mon.a (mon.0) 1137 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2023-07-10T17:18:20.829 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.707684+0000 mon.a (mon.0) 1138 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:20.829 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.708687+0000 mon.a (mon.0) 1139 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:20.829 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.709685+0000 mon.a (mon.0) 1140 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:20.830 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.710717+0000 mon.a (mon.0) 1141 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:20.831 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.711673+0000 mon.a (mon.0) 1142 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:20.831 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.712721+0000 mon.a (mon.0) 1143 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:20.831 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[50362]: audit 2023-07-10T17:18:20.713815+0000 mon.a (mon.0) 1144 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2023-07-10T17:18:21.079 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.840641196Z level=info msg="Executing migration" id="increase max description length to 2048" 2023-07-10T17:18:21.080 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.882599548Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2023-07-10T17:18:21.080 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.907898831Z level=info msg="Executing migration" id="create data_keys table" 2023-07-10T17:18:21.080 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.931305853Z level=info msg="Executing migration" id="create secrets table" 2023-07-10T17:18:21.080 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:20 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:20.956560643Z level=info msg="Executing migration" id="rename data_keys name column to id" 2023-07-10T17:18:21.080 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.04853693Z level=info msg="Executing migration" id="add name column into data_keys" 2023-07-10T17:18:21.081 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.073792476Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2023-07-10T17:18:21.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: cluster 2023-07-10T17:18:19.334274+0000 mgr.y (mgr.25390) 483 : cluster 0 pgmap v346: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:21.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.416762+0000 mon.a (mon.0) 1092 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:21.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.430317+0000 mon.a (mon.0) 1093 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:21.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.432060+0000 mon.a (mon.0) 1094 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:18:21.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.433919+0000 mon.a (mon.0) 1095 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:18:21.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.445597+0000 mon.a (mon.0) 1096 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:21.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.529776+0000 mon.a (mon.0) 1097 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:18:21.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.538808+0000 mon.a (mon.0) 1098 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:18:21.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.543970+0000 mon.a (mon.0) 1099 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.546957+0000 mon.a (mon.0) 1100 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.549244+0000 mon.a (mon.0) 1101 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.149 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.552920+0000 mon.a (mon.0) 1102 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.149 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.557172+0000 mon.a (mon.0) 1103 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.150 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.560196+0000 mon.a (mon.0) 1104 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.150 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.562316+0000 mon.a (mon.0) 1105 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.150 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.564347+0000 mon.a (mon.0) 1106 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.150 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.567345+0000 mon.a (mon.0) 1107 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.151 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.570675+0000 mon.a (mon.0) 1108 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.151 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.573491+0000 mon.a (mon.0) 1109 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.152 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.577173+0000 mon.a (mon.0) 1110 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.152 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: cluster 2023-07-10T17:18:19.334274+0000 mgr.y (mgr.25390) 483 : cluster 0 pgmap v346: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:21.152 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.416762+0000 mon.a (mon.0) 1092 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:21.152 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.430317+0000 mon.a (mon.0) 1093 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:21.153 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.432060+0000 mon.a (mon.0) 1094 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:18:21.153 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.433919+0000 mon.a (mon.0) 1095 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:18:21.153 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.445597+0000 mon.a (mon.0) 1096 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:21.153 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.529776+0000 mon.a (mon.0) 1097 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:18:21.153 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.538808+0000 mon.a (mon.0) 1098 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-10T17:18:21.153 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.543970+0000 mon.a (mon.0) 1099 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.153 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.546957+0000 mon.a (mon.0) 1100 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.154 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.549244+0000 mon.a (mon.0) 1101 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.154 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.552920+0000 mon.a (mon.0) 1102 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.154 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.557172+0000 mon.a (mon.0) 1103 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.154 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.560196+0000 mon.a (mon.0) 1104 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.154 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.562316+0000 mon.a (mon.0) 1105 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.154 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.564347+0000 mon.a (mon.0) 1106 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.154 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.567345+0000 mon.a (mon.0) 1107 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.154 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.570675+0000 mon.a (mon.0) 1108 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.155 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.573491+0000 mon.a (mon.0) 1109 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.155 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.577173+0000 mon.a (mon.0) 1110 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.580397+0000 mon.a (mon.0) 1111 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.583472+0000 mon.a (mon.0) 1112 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.586600+0000 mon.a (mon.0) 1113 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.589450+0000 mon.a (mon.0) 1114 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.592204+0000 mon.a (mon.0) 1115 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.605786+0000 mon.a (mon.0) 1116 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:21.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.607590+0000 mon.a (mon.0) 1117 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2023-07-10T17:18:21.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.615397+0000 mon.a (mon.0) 1118 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2023-07-10T17:18:21.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.616904+0000 mon.a (mon.0) 1119 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:21.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.626225+0000 mon.a (mon.0) 1120 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2023-07-10T17:18:21.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.627605+0000 mon.a (mon.0) 1121 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2023-07-10T17:18:21.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.635443+0000 mon.a (mon.0) 1122 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2023-07-10T17:18:21.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.637201+0000 mon.a (mon.0) 1123 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2023-07-10T17:18:21.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.645982+0000 mon.a (mon.0) 1124 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2023-07-10T17:18:21.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.647758+0000 mon.a (mon.0) 1125 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2023-07-10T17:18:21.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.656156+0000 mon.a (mon.0) 1126 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2023-07-10T17:18:21.159 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.657484+0000 mon.a (mon.0) 1127 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2023-07-10T17:18:21.159 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.580397+0000 mon.a (mon.0) 1111 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.159 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.583472+0000 mon.a (mon.0) 1112 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.160 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.586600+0000 mon.a (mon.0) 1113 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.160 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.589450+0000 mon.a (mon.0) 1114 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.160 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.592204+0000 mon.a (mon.0) 1115 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:21.160 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.605786+0000 mon.a (mon.0) 1116 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:21.161 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.607590+0000 mon.a (mon.0) 1117 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2023-07-10T17:18:21.161 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.615397+0000 mon.a (mon.0) 1118 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2023-07-10T17:18:21.161 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.616904+0000 mon.a (mon.0) 1119 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:21.162 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.626225+0000 mon.a (mon.0) 1120 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2023-07-10T17:18:21.162 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.627605+0000 mon.a (mon.0) 1121 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2023-07-10T17:18:21.163 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.635443+0000 mon.a (mon.0) 1122 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2023-07-10T17:18:21.163 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.637201+0000 mon.a (mon.0) 1123 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2023-07-10T17:18:21.163 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.645982+0000 mon.a (mon.0) 1124 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2023-07-10T17:18:21.163 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.647758+0000 mon.a (mon.0) 1125 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2023-07-10T17:18:21.164 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.656156+0000 mon.a (mon.0) 1126 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2023-07-10T17:18:21.164 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.657484+0000 mon.a (mon.0) 1127 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2023-07-10T17:18:21.166 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.665948+0000 mon.a (mon.0) 1128 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2023-07-10T17:18:21.168 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.667495+0000 mon.a (mon.0) 1129 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2023-07-10T17:18:21.168 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.675711+0000 mon.a (mon.0) 1130 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2023-07-10T17:18:21.169 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.676910+0000 mon.a (mon.0) 1131 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:21.169 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.678007+0000 mon.a (mon.0) 1132 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2023-07-10T17:18:21.169 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.685818+0000 mon.a (mon.0) 1133 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2023-07-10T17:18:21.169 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.687101+0000 mon.a (mon.0) 1134 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2023-07-10T17:18:21.169 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.695800+0000 mon.a (mon.0) 1135 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2023-07-10T17:18:21.169 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.697425+0000 mon.a (mon.0) 1136 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2023-07-10T17:18:21.170 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.706355+0000 mon.a (mon.0) 1137 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2023-07-10T17:18:21.170 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.707684+0000 mon.a (mon.0) 1138 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:21.170 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.708687+0000 mon.a (mon.0) 1139 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:21.170 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.709685+0000 mon.a (mon.0) 1140 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:21.171 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.710717+0000 mon.a (mon.0) 1141 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:21.171 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.711673+0000 mon.a (mon.0) 1142 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:21.171 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.712721+0000 mon.a (mon.0) 1143 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:21.171 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[63900]: audit 2023-07-10T17:18:20.713815+0000 mon.a (mon.0) 1144 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2023-07-10T17:18:21.173 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.665948+0000 mon.a (mon.0) 1128 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2023-07-10T17:18:21.173 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.667495+0000 mon.a (mon.0) 1129 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2023-07-10T17:18:21.174 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.675711+0000 mon.a (mon.0) 1130 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2023-07-10T17:18:21.174 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.676910+0000 mon.a (mon.0) 1131 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:21.174 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.678007+0000 mon.a (mon.0) 1132 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2023-07-10T17:18:21.174 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.685818+0000 mon.a (mon.0) 1133 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2023-07-10T17:18:21.174 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.687101+0000 mon.a (mon.0) 1134 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2023-07-10T17:18:21.174 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.695800+0000 mon.a (mon.0) 1135 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2023-07-10T17:18:21.174 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.697425+0000 mon.a (mon.0) 1136 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2023-07-10T17:18:21.175 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.706355+0000 mon.a (mon.0) 1137 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2023-07-10T17:18:21.175 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.707684+0000 mon.a (mon.0) 1138 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:21.175 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.708687+0000 mon.a (mon.0) 1139 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:21.175 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.709685+0000 mon.a (mon.0) 1140 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:21.175 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.710717+0000 mon.a (mon.0) 1141 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:21.175 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.711673+0000 mon.a (mon.0) 1142 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:21.175 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.712721+0000 mon.a (mon.0) 1143 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2023-07-10T17:18:21.176 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:20 smithi116 bash[65980]: audit 2023-07-10T17:18:20.713815+0000 mon.a (mon.0) 1144 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2023-07-10T17:18:21.349 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.115779759Z level=info msg="Executing migration" id="rename data_keys name column to label" 2023-07-10T17:18:21.349 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.166093515Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2023-07-10T17:18:21.350 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.233108071Z level=info msg="Executing migration" id="create kv_store table v1" 2023-07-10T17:18:21.350 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.258437479Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2023-07-10T17:18:21.350 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.283653766Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2023-07-10T17:18:21.351 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.308848507Z level=info msg="Executing migration" id="create permission table" 2023-07-10T17:18:21.351 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.342702354Z level=info msg="Executing migration" id="add unique index permission.role_id" 2023-07-10T17:18:21.596 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.367977729Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2023-07-10T17:18:21.596 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.393269675Z level=info msg="Executing migration" id="create role table" 2023-07-10T17:18:21.596 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.418325952Z level=info msg="Executing migration" id="add column display_name" 2023-07-10T17:18:21.597 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.443587776Z level=info msg="Executing migration" id="add column group_name" 2023-07-10T17:18:21.597 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.527289223Z level=info msg="Executing migration" id="add index role.org_id" 2023-07-10T17:18:21.597 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.550733811Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2023-07-10T17:18:21.597 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.592982498Z level=info msg="Executing migration" id="add index role_org_id_uid" 2023-07-10T17:18:21.848 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.618221955Z level=info msg="Executing migration" id="create team role table" 2023-07-10T17:18:21.849 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.643490137Z level=info msg="Executing migration" id="add index team_role.org_id" 2023-07-10T17:18:21.849 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.668750691Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2023-07-10T17:18:21.849 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.719114049Z level=info msg="Executing migration" id="add index team_role.team_id" 2023-07-10T17:18:21.849 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.761255689Z level=info msg="Executing migration" id="create user role table" 2023-07-10T17:18:21.849 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.78641101Z level=info msg="Executing migration" id="add index user_role.org_id" 2023-07-10T17:18:21.849 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.844578982Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2023-07-10T17:18:21.850 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[50362]: cephadm 2023-07-10T17:18:20.435498+0000 mgr.y (mgr.25390) 484 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:18:21.850 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[50362]: audit 2023-07-10T17:18:20.530670+0000 mgr.y (mgr.25390) 485 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:18:21.850 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[50362]: cephadm 2023-07-10T17:18:20.593855+0000 mgr.y (mgr.25390) 486 : cephadm 1 Upgrade: Finalizing container_image settings 2023-07-10T17:18:21.850 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[50362]: audit 2023-07-10T17:18:20.722683+0000 mon.a (mon.0) 1145 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2023-07-10T17:18:21.850 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[50362]: cephadm 2023-07-10T17:18:20.724620+0000 mgr.y (mgr.25390) 487 : cephadm 1 Upgrade: Complete! 2023-07-10T17:18:21.850 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[50362]: audit 2023-07-10T17:18:20.725568+0000 mon.a (mon.0) 1146 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2023-07-10T17:18:21.850 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[50362]: audit 2023-07-10T17:18:20.741215+0000 mon.a (mon.0) 1147 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2023-07-10T17:18:21.851 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[50362]: audit 2023-07-10T17:18:20.742664+0000 mon.a (mon.0) 1148 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:18:21.851 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[50362]: audit 2023-07-10T17:18:20.743800+0000 mon.a (mon.0) 1149 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:18:21.851 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[50362]: cephadm 2023-07-10T17:18:20.744721+0000 mgr.y (mgr.25390) 488 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:18:21.851 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[50362]: audit 2023-07-10T17:18:20.750579+0000 mon.a (mon.0) 1150 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:21.851 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[50362]: audit 2023-07-10T17:18:20.835231+0000 mon.a (mon.0) 1151 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:18:21.851 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[50362]: audit 2023-07-10T17:18:20.836896+0000 mon.a (mon.0) 1152 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:18:21.851 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[50362]: cephadm 2023-07-10T17:18:20.838539+0000 mgr.y (mgr.25390) 489 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:18:21.852 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[50362]: audit 2023-07-10T17:18:20.847810+0000 mon.a (mon.0) 1153 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:21.852 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[50362]: audit 2023-07-10T17:18:21.269365+0000 mgr.y (mgr.25390) 490 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:22.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.887140478Z level=info msg="Executing migration" id="add index user_role.user_id" 2023-07-10T17:18:22.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.912405434Z level=info msg="Executing migration" id="create builtin role table" 2023-07-10T17:18:22.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:21 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:21.996053982Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2023-07-10T17:18:22.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.038010523Z level=info msg="Executing migration" id="add index builtin_role.name" 2023-07-10T17:18:22.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.063205601Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2023-07-10T17:18:22.108 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.103710564Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2023-07-10T17:18:22.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[63900]: cephadm 2023-07-10T17:18:20.435498+0000 mgr.y (mgr.25390) 484 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:18:22.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[63900]: audit 2023-07-10T17:18:20.530670+0000 mgr.y (mgr.25390) 485 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:18:22.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[63900]: cephadm 2023-07-10T17:18:20.593855+0000 mgr.y (mgr.25390) 486 : cephadm 1 Upgrade: Finalizing container_image settings 2023-07-10T17:18:22.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[63900]: audit 2023-07-10T17:18:20.722683+0000 mon.a (mon.0) 1145 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2023-07-10T17:18:22.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[63900]: cephadm 2023-07-10T17:18:20.724620+0000 mgr.y (mgr.25390) 487 : cephadm 1 Upgrade: Complete! 2023-07-10T17:18:22.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[63900]: audit 2023-07-10T17:18:20.725568+0000 mon.a (mon.0) 1146 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2023-07-10T17:18:22.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[63900]: audit 2023-07-10T17:18:20.741215+0000 mon.a (mon.0) 1147 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2023-07-10T17:18:22.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[63900]: audit 2023-07-10T17:18:20.742664+0000 mon.a (mon.0) 1148 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:18:22.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[63900]: audit 2023-07-10T17:18:20.743800+0000 mon.a (mon.0) 1149 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:18:22.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[63900]: cephadm 2023-07-10T17:18:20.744721+0000 mgr.y (mgr.25390) 488 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:18:22.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[63900]: audit 2023-07-10T17:18:20.750579+0000 mon.a (mon.0) 1150 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:22.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[63900]: audit 2023-07-10T17:18:20.835231+0000 mon.a (mon.0) 1151 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:18:22.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[63900]: audit 2023-07-10T17:18:20.836896+0000 mon.a (mon.0) 1152 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:18:22.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[63900]: cephadm 2023-07-10T17:18:20.838539+0000 mgr.y (mgr.25390) 489 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:18:22.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[63900]: audit 2023-07-10T17:18:20.847810+0000 mon.a (mon.0) 1153 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:22.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[63900]: audit 2023-07-10T17:18:21.269365+0000 mgr.y (mgr.25390) 490 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:22.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[65980]: cephadm 2023-07-10T17:18:20.435498+0000 mgr.y (mgr.25390) 484 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:18:22.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[65980]: audit 2023-07-10T17:18:20.530670+0000 mgr.y (mgr.25390) 485 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-10T17:18:22.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[65980]: cephadm 2023-07-10T17:18:20.593855+0000 mgr.y (mgr.25390) 486 : cephadm 1 Upgrade: Finalizing container_image settings 2023-07-10T17:18:22.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[65980]: audit 2023-07-10T17:18:20.722683+0000 mon.a (mon.0) 1145 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2023-07-10T17:18:22.148 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[65980]: cephadm 2023-07-10T17:18:20.724620+0000 mgr.y (mgr.25390) 487 : cephadm 1 Upgrade: Complete! 2023-07-10T17:18:22.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[65980]: audit 2023-07-10T17:18:20.725568+0000 mon.a (mon.0) 1146 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2023-07-10T17:18:22.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[65980]: audit 2023-07-10T17:18:20.741215+0000 mon.a (mon.0) 1147 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2023-07-10T17:18:22.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[65980]: audit 2023-07-10T17:18:20.742664+0000 mon.a (mon.0) 1148 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:18:22.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[65980]: audit 2023-07-10T17:18:20.743800+0000 mon.a (mon.0) 1149 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:18:22.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[65980]: cephadm 2023-07-10T17:18:20.744721+0000 mgr.y (mgr.25390) 488 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:18:22.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[65980]: audit 2023-07-10T17:18:20.750579+0000 mon.a (mon.0) 1150 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:22.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[65980]: audit 2023-07-10T17:18:20.835231+0000 mon.a (mon.0) 1151 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:18:22.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[65980]: audit 2023-07-10T17:18:20.836896+0000 mon.a (mon.0) 1152 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:18:22.150 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[65980]: cephadm 2023-07-10T17:18:20.838539+0000 mgr.y (mgr.25390) 489 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:18:22.151 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[65980]: audit 2023-07-10T17:18:20.847810+0000 mon.a (mon.0) 1153 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:22.151 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:21 smithi116 bash[65980]: audit 2023-07-10T17:18:21.269365+0000 mgr.y (mgr.25390) 490 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:22.377 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.145785549Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2023-07-10T17:18:22.377 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.187704352Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2023-07-10T17:18:22.378 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.212956338Z level=info msg="Executing migration" id="add unique index role.uid" 2023-07-10T17:18:22.378 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.254892713Z level=info msg="Executing migration" id="create seed assignment table" 2023-07-10T17:18:22.378 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.297155333Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2023-07-10T17:18:22.378 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.322191306Z level=info msg="Executing migration" id="add column hidden to role table" 2023-07-10T17:18:22.378 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.347432929Z level=info msg="Executing migration" id="create query_history table v1" 2023-07-10T17:18:22.378 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.372816739Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2023-07-10T17:18:22.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.414801344Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2023-07-10T17:18:22.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.4732963Z level=info msg="Executing migration" id="teams permissions migration" 2023-07-10T17:18:22.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.498567006Z level=info msg="Executing migration" id="dashboard permissions" 2023-07-10T17:18:22.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.532183277Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2023-07-10T17:18:22.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.557444436Z level=info msg="Executing migration" id="drop managed folder create actions" 2023-07-10T17:18:22.638 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.60628784Z level=info msg="Executing migration" id="alerting notification permissions" 2023-07-10T17:18:22.970 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[50362]: cluster 2023-07-10T17:18:21.335650+0000 mgr.y (mgr.25390) 491 : cluster 0 pgmap v347: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:22.971 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[50362]: audit 2023-07-10T17:18:21.743416+0000 mon.a (mon.0) 1154 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:22.971 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.648387852Z level=info msg="Executing migration" id="create query_history_star table v1" 2023-07-10T17:18:22.971 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.673764656Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2023-07-10T17:18:22.972 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.724088319Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2023-07-10T17:18:22.972 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.765992839Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2023-07-10T17:18:22.972 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.791373639Z level=info msg="Executing migration" id="create correlation table v1" 2023-07-10T17:18:22.972 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.816627898Z level=info msg="Executing migration" id="add index correlations.uid" 2023-07-10T17:18:22.972 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.841863461Z level=info msg="Executing migration" id="add index correlations.source_uid" 2023-07-10T17:18:22.972 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.883847999Z level=info msg="Executing migration" id="add correlation config column" 2023-07-10T17:18:22.972 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:22 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:22.967461767Z level=info msg="Executing migration" id="create entity_events table" 2023-07-10T17:18:23.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:22 smithi116 bash[65980]: cluster 2023-07-10T17:18:21.335650+0000 mgr.y (mgr.25390) 491 : cluster 0 pgmap v347: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:23.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:22 smithi116 bash[65980]: audit 2023-07-10T17:18:21.743416+0000 mon.a (mon.0) 1154 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:23.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:22 smithi116 bash[63900]: cluster 2023-07-10T17:18:21.335650+0000 mgr.y (mgr.25390) 491 : cluster 0 pgmap v347: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:23.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:22 smithi116 bash[63900]: audit 2023-07-10T17:18:21.743416+0000 mon.a (mon.0) 1154 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:18:23.223 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.009461381Z level=info msg="Executing migration" id="create dashboard public config v1" 2023-07-10T17:18:23.223 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.05175471Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2023-07-10T17:18:23.223 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.05212888Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2023-07-10T17:18:23.223 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.101424812Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2023-07-10T17:18:23.223 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.101867521Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2023-07-10T17:18:23.224 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.143686584Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2023-07-10T17:18:23.224 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.168936443Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2023-07-10T17:18:23.224 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.19417422Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2023-07-10T17:18:23.224 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.219475232Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2023-07-10T17:18:23.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.244679561Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2023-07-10T17:18:23.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.269928121Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2023-07-10T17:18:23.486 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.311897947Z level=info msg="Executing migration" id="Drop public config table" 2023-07-10T17:18:23.486 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.3539467Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2023-07-10T17:18:23.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.429230162Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2023-07-10T17:18:23.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.456230361Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2023-07-10T17:18:23.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.481440598Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2023-07-10T17:18:23.745 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.506701647Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2023-07-10T17:18:23.745 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.540242829Z level=info msg="Executing migration" id="add annotations_enabled column" 2023-07-10T17:18:23.745 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.598956616Z level=info msg="Executing migration" id="add time_selection_enabled column" 2023-07-10T17:18:23.746 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.640943763Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2023-07-10T17:18:23.746 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.666182739Z level=info msg="Executing migration" id="add share column" 2023-07-10T17:18:23.746 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.691478948Z level=info msg="Executing migration" id="create default alerting folders" 2023-07-10T17:18:23.746 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.716684977Z level=info msg="Executing migration" id="create file table" 2023-07-10T17:18:24.022 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.742089277Z level=info msg="Executing migration" id="file table idx: path natural pk" 2023-07-10T17:18:24.022 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.783951639Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2023-07-10T17:18:24.023 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.817696394Z level=info msg="Executing migration" id="create file_meta table" 2023-07-10T17:18:24.023 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.842926784Z level=info msg="Executing migration" id="file table idx: path key" 2023-07-10T17:18:24.023 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.92656143Z level=info msg="Executing migration" id="set path collation in file table" 2023-07-10T17:18:24.023 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.951797243Z level=info msg="Executing migration" id="managed permissions migration" 2023-07-10T17:18:24.023 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:23 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:23.977059202Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2023-07-10T17:18:24.023 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.019054225Z level=info msg="Executing migration" id="RBAC action name migrator" 2023-07-10T17:18:24.282 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.044333114Z level=info msg="Executing migration" id="Add UID column to playlist" 2023-07-10T17:18:24.283 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.069567006Z level=info msg="Executing migration" id="Update uid column values in playlist" 2023-07-10T17:18:24.283 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.11148413Z level=info msg="Executing migration" id="Add index for uid in playlist" 2023-07-10T17:18:24.283 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.153549962Z level=info msg="Executing migration" id="update group index for alert rules" 2023-07-10T17:18:24.283 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.194832254Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2023-07-10T17:18:24.283 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.237124055Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2023-07-10T17:18:24.283 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.279314767Z level=info msg="Executing migration" id="add action column to seed_assignment" 2023-07-10T17:18:24.552 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.304855241Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2023-07-10T17:18:24.552 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.380179325Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2023-07-10T17:18:24.552 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.40552355Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2023-07-10T17:18:24.553 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.45569026Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2023-07-10T17:18:24.553 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.481062617Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2023-07-10T17:18:24.554 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.506299686Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2023-07-10T17:18:24.554 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.548430126Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2023-07-10T17:18:24.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[50362]: cluster 2023-07-10T17:18:23.336833+0000 mgr.y (mgr.25390) 492 : cluster 0 pgmap v348: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:24.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.590170924Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2023-07-10T17:18:24.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.631714614Z level=info msg="Executing migration" id="create folder table" 2023-07-10T17:18:24.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.67416457Z level=info msg="Executing migration" id="Add index for parent_uid" 2023-07-10T17:18:24.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.699472624Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2023-07-10T17:18:24.887 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.724631158Z level=info msg="Executing migration" id="Update folder title length" 2023-07-10T17:18:24.888 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.766643514Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2023-07-10T17:18:24.888 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=migrator t=2023-07-10T17:18:24.791892793Z level=info msg="migrations completed" performed=485 skipped=0 duration=21.980177466s 2023-07-10T17:18:24.888 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=sqlstore t=2023-07-10T17:18:24.793369781Z level=info msg="Created default organization" 2023-07-10T17:18:25.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:24 smithi116 bash[65980]: cluster 2023-07-10T17:18:23.336833+0000 mgr.y (mgr.25390) 492 : cluster 0 pgmap v348: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:25.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:24 smithi116 bash[63900]: cluster 2023-07-10T17:18:23.336833+0000 mgr.y (mgr.25390) 492 : cluster 0 pgmap v348: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:25.387 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=plugin.loader t=2023-07-10T17:18:24.943707103Z level=info msg="Plugin registered" pluginID=input 2023-07-10T17:18:25.388 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=plugin.loader t=2023-07-10T17:18:24.972771777Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2023-07-10T17:18:25.388 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=plugin.loader t=2023-07-10T17:18:24.972801148Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2023-07-10T17:18:25.388 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=secrets t=2023-07-10T17:18:24.972864434Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2023-07-10T17:18:25.389 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=query_data t=2023-07-10T17:18:24.974828061Z level=info msg="Query Service initialization" 2023-07-10T17:18:25.389 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:24 smithi192 bash[68262]: logger=live.push_http t=2023-07-10T17:18:24.979107209Z level=info msg="Live Push Gateway initialization" 2023-07-10T17:18:26.269 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:25 smithi192 bash[68262]: logger=infra.usagestats.collector t=2023-07-10T17:18:25.935682215Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2023-07-10T17:18:26.270 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:25 smithi192 bash[68262]: logger=provisioning.datasources t=2023-07-10T17:18:25.994224065Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2023-07-10T17:18:26.270 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:26 smithi192 bash[68262]: logger=provisioning.datasources t=2023-07-10T17:18:26.070413331Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2023-07-10T17:18:26.270 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:26 smithi192 bash[68262]: logger=provisioning.alerting t=2023-07-10T17:18:26.144295931Z level=info msg="starting to provision alerting" 2023-07-10T17:18:26.270 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:26 smithi192 bash[68262]: logger=provisioning.alerting t=2023-07-10T17:18:26.144363367Z level=info msg="finished to provision alerting" 2023-07-10T17:18:26.270 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:26 smithi192 bash[68262]: logger=grafanaStorageLogger t=2023-07-10T17:18:26.145154339Z level=info msg="storage starting" 2023-07-10T17:18:26.270 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:26 smithi192 bash[68262]: logger=ngalert.state.manager t=2023-07-10T17:18:26.145240955Z level=info msg="Warming state cache for startup" 2023-07-10T17:18:26.271 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:26 smithi192 bash[68262]: logger=http.server t=2023-07-10T17:18:26.155451506Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2023-07-10T17:18:26.271 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:26 smithi192 bash[68262]: logger=sqlstore.transactions t=2023-07-10T17:18:26.15673745Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2023-07-10T17:18:26.271 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:26 smithi192 bash[68262]: logger=ngalert.state.manager t=2023-07-10T17:18:26.265481831Z level=info msg="State cache has been initialized" states=0 duration=120.239719ms 2023-07-10T17:18:26.271 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:26 smithi192 bash[68262]: logger=ngalert.multiorg.alertmanager t=2023-07-10T17:18:26.265575433Z level=info msg="starting MultiOrg Alertmanager" 2023-07-10T17:18:26.271 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:26 smithi192 bash[68262]: logger=ticker t=2023-07-10T17:18:26.265636054Z level=info msg=starting first_tick=2023-07-10T17:18:30Z 2023-07-10T17:18:26.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:26 smithi192 bash[68262]: logger=sqlstore.transactions t=2023-07-10T17:18:26.409545663Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2023-07-10T17:18:26.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:26 smithi192 bash[68262]: logger=sqlstore.transactions t=2023-07-10T17:18:26.420733249Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2023-07-10T17:18:26.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:26 smithi192 bash[68262]: logger=sqlstore.transactions t=2023-07-10T17:18:26.485381822Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2023-07-10T17:18:26.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:26 smithi192 bash[68262]: logger=sqlstore.transactions t=2023-07-10T17:18:26.496560854Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2023-07-10T17:18:26.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:18:26 smithi192 bash[68262]: logger=sqlstore.transactions t=2023-07-10T17:18:26.507634112Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=4 code="database is locked" 2023-07-10T17:18:27.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:26 smithi192 bash[50362]: cluster 2023-07-10T17:18:25.337944+0000 mgr.y (mgr.25390) 493 : cluster 0 pgmap v349: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:27.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:26 smithi116 bash[65980]: cluster 2023-07-10T17:18:25.337944+0000 mgr.y (mgr.25390) 493 : cluster 0 pgmap v349: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:27.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:26 smithi116 bash[63900]: cluster 2023-07-10T17:18:25.337944+0000 mgr.y (mgr.25390) 493 : cluster 0 pgmap v349: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:29.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:28 smithi192 bash[50362]: cluster 2023-07-10T17:18:27.339489+0000 mgr.y (mgr.25390) 494 : cluster 0 pgmap v350: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:29.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:28 smithi116 bash[65980]: cluster 2023-07-10T17:18:27.339489+0000 mgr.y (mgr.25390) 494 : cluster 0 pgmap v350: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:29.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:28 smithi116 bash[63900]: cluster 2023-07-10T17:18:27.339489+0000 mgr.y (mgr.25390) 494 : cluster 0 pgmap v350: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:31.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:30 smithi192 bash[50362]: cluster 2023-07-10T17:18:29.340379+0000 mgr.y (mgr.25390) 495 : cluster 0 pgmap v351: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:31.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:30 smithi116 bash[63900]: cluster 2023-07-10T17:18:29.340379+0000 mgr.y (mgr.25390) 495 : cluster 0 pgmap v351: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:31.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:30 smithi116 bash[65980]: cluster 2023-07-10T17:18:29.340379+0000 mgr.y (mgr.25390) 495 : cluster 0 pgmap v351: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:32.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:31 smithi192 bash[50362]: audit 2023-07-10T17:18:31.279815+0000 mgr.y (mgr.25390) 496 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:32.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:31 smithi116 bash[63900]: audit 2023-07-10T17:18:31.279815+0000 mgr.y (mgr.25390) 496 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:32.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:31 smithi116 bash[65980]: audit 2023-07-10T17:18:31.279815+0000 mgr.y (mgr.25390) 496 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:33.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:32 smithi192 bash[50362]: cluster 2023-07-10T17:18:31.341719+0000 mgr.y (mgr.25390) 497 : cluster 0 pgmap v352: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:33.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:32 smithi116 bash[63900]: cluster 2023-07-10T17:18:31.341719+0000 mgr.y (mgr.25390) 497 : cluster 0 pgmap v352: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:33.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:32 smithi116 bash[65980]: cluster 2023-07-10T17:18:31.341719+0000 mgr.y (mgr.25390) 497 : cluster 0 pgmap v352: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:35.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:34 smithi192 bash[50362]: cluster 2023-07-10T17:18:33.342964+0000 mgr.y (mgr.25390) 498 : cluster 0 pgmap v353: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:35.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:34 smithi116 bash[65980]: cluster 2023-07-10T17:18:33.342964+0000 mgr.y (mgr.25390) 498 : cluster 0 pgmap v353: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:35.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:34 smithi116 bash[63900]: cluster 2023-07-10T17:18:33.342964+0000 mgr.y (mgr.25390) 498 : cluster 0 pgmap v353: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:36.243 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph orch ps' 2023-07-10T17:18:37.141 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:36 smithi192 bash[50362]: cluster 2023-07-10T17:18:35.344025+0000 mgr.y (mgr.25390) 499 : cluster 0 pgmap v354: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:37.142 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:36 smithi192 bash[50362]: audit 2023-07-10T17:18:35.352265+0000 mgr.y (mgr.25390) 500 : audit 0 from='client.35199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:37.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:36 smithi116 bash[63900]: cluster 2023-07-10T17:18:35.344025+0000 mgr.y (mgr.25390) 499 : cluster 0 pgmap v354: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:37.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:36 smithi116 bash[63900]: audit 2023-07-10T17:18:35.352265+0000 mgr.y (mgr.25390) 500 : audit 0 from='client.35199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:37.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:36 smithi116 bash[65980]: cluster 2023-07-10T17:18:35.344025+0000 mgr.y (mgr.25390) 499 : cluster 0 pgmap v354: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:37.147 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:36 smithi116 bash[65980]: audit 2023-07-10T17:18:35.352265+0000 mgr.y (mgr.25390) 500 : audit 0 from='client.35199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:39.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:38 smithi192 bash[50362]: cluster 2023-07-10T17:18:37.345452+0000 mgr.y (mgr.25390) 501 : cluster 0 pgmap v355: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:39.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:38 smithi116 bash[63900]: cluster 2023-07-10T17:18:37.345452+0000 mgr.y (mgr.25390) 501 : cluster 0 pgmap v355: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:39.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:38 smithi116 bash[65980]: cluster 2023-07-10T17:18:37.345452+0000 mgr.y (mgr.25390) 501 : cluster 0 pgmap v355: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:40.003 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:18:40.004 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (64s) 56s ago 18m 15.6M - 0.25.0 c8568f914cd2 5032fe494a7b 2023-07-10T17:18:40.004 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (38s) 32s ago 17m 25.1M - 9.4.7 2c41d148cca3 cb0b03556cc9 2023-07-10T17:18:40.004 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 running (2m) 56s ago 15m 48.0M - 3.6 17e8f0d79247 fbdecd8618b7 2023-07-10T17:18:40.004 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (9m) 32s ago 22m 431M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:18:40.004 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (10m) 56s ago 24m 487M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:18:40.004 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (9m) 56s ago 24m 84.1M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:18:40.005 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (8m) 32s ago 22m 62.5M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 74a082a2e795 2023-07-10T17:18:40.005 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (8m) 56s ago 22m 63.0M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:18:40.005 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (115s) 56s ago 18m 9140k - 1.5.0 0da6a335fe13 64b482421ffb 2023-07-10T17:18:40.005 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (111s) 32s ago 18m 9744k - 1.5.0 0da6a335fe13 82b166142066 2023-07-10T17:18:40.005 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (6m) 56s ago 22m 63.7M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 0562f587fbca 2023-07-10T17:18:40.005 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (6m) 56s ago 21m 63.5M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 5b2a12a0bfc9 2023-07-10T17:18:40.005 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (5m) 56s ago 21m 56.1M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 88096bc0faa9 2023-07-10T17:18:40.005 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (5m) 56s ago 20m 61.2M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 61ae280c471b 2023-07-10T17:18:40.006 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (4m) 32s ago 20m 62.3M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 44d809c18118 2023-07-10T17:18:40.006 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (4m) 32s ago 19m 58.3M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 179270eb7044 2023-07-10T17:18:40.006 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (3m) 32s ago 19m 55.4M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 b680bfbe9a0c 2023-07-10T17:18:40.006 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (3m) 32s ago 19m 57.4M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 bd3788803e68 2023-07-10T17:18:40.006 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 running (89s) 32s ago 18m 37.0M - 2.43.0 a07b618ecd1d 48c15c1430f7 2023-07-10T17:18:40.006 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (3m) 56s ago 16m 84.3M - 18.1.2-30-g8507faa7 17e8f0d79247 ef72a4e0ae29 2023-07-10T17:18:40.006 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (2m) 32s ago 16m 84.7M - 18.1.2-30-g8507faa7 17e8f0d79247 59b0f0b41f44 2023-07-10T17:18:40.007 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (2m) 56s ago 15m 84.4M - 18.1.2-30-g8507faa7 17e8f0d79247 42f4f263b75e 2023-07-10T17:18:40.007 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (2m) 32s ago 15m 84.3M - 18.1.2-30-g8507faa7 17e8f0d79247 2a257a88c4e9 2023-07-10T17:18:40.812 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph versions' 2023-07-10T17:18:41.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:40 smithi192 bash[50362]: cluster 2023-07-10T17:18:39.346410+0000 mgr.y (mgr.25390) 502 : cluster 0 pgmap v356: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:41.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:40 smithi192 bash[50362]: audit 2023-07-10T17:18:39.974255+0000 mgr.y (mgr.25390) 503 : audit 0 from='client.35205 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:41.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:40 smithi116 bash[63900]: cluster 2023-07-10T17:18:39.346410+0000 mgr.y (mgr.25390) 502 : cluster 0 pgmap v356: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:41.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:40 smithi116 bash[63900]: audit 2023-07-10T17:18:39.974255+0000 mgr.y (mgr.25390) 503 : audit 0 from='client.35205 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:41.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:40 smithi116 bash[65980]: cluster 2023-07-10T17:18:39.346410+0000 mgr.y (mgr.25390) 502 : cluster 0 pgmap v356: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:41.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:40 smithi116 bash[65980]: audit 2023-07-10T17:18:39.974255+0000 mgr.y (mgr.25390) 503 : audit 0 from='client.35205 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:18:42.060 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:41 smithi116 bash[65980]: audit 2023-07-10T17:18:41.291038+0000 mgr.y (mgr.25390) 504 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:42.060 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:41 smithi116 bash[63900]: audit 2023-07-10T17:18:41.291038+0000 mgr.y (mgr.25390) 504 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:42.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:41 smithi192 bash[50362]: audit 2023-07-10T17:18:41.291038+0000 mgr.y (mgr.25390) 504 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:43.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:42 smithi192 bash[50362]: cluster 2023-07-10T17:18:41.347705+0000 mgr.y (mgr.25390) 505 : cluster 0 pgmap v357: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:43.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:42 smithi116 bash[63900]: cluster 2023-07-10T17:18:41.347705+0000 mgr.y (mgr.25390) 505 : cluster 0 pgmap v357: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:43.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:42 smithi116 bash[65980]: cluster 2023-07-10T17:18:41.347705+0000 mgr.y (mgr.25390) 505 : cluster 0 pgmap v357: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:44.527 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:18:44.528 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:18:44.528 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:18:44.528 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:18:44.529 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:18:44.529 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:18:44.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:18:44.529 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:18:44.530 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 8 2023-07-10T17:18:44.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:18:44.530 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:18:44.530 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 4 2023-07-10T17:18:44.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:18:44.532 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:18:44.532 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 17 2023-07-10T17:18:44.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:18:44.533 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:18:45.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:44 smithi192 bash[50362]: cluster 2023-07-10T17:18:43.348857+0000 mgr.y (mgr.25390) 506 : cluster 0 pgmap v358: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:45.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:44 smithi192 bash[50362]: audit 2023-07-10T17:18:44.521813+0000 mon.c (mon.1) 38 : audit 0 from='client.45073 172.21.15.116:0/2189054938' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:45.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:44 smithi116 bash[63900]: cluster 2023-07-10T17:18:43.348857+0000 mgr.y (mgr.25390) 506 : cluster 0 pgmap v358: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:45.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:44 smithi116 bash[63900]: audit 2023-07-10T17:18:44.521813+0000 mon.c (mon.1) 38 : audit 0 from='client.45073 172.21.15.116:0/2189054938' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:45.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:44 smithi116 bash[65980]: cluster 2023-07-10T17:18:43.348857+0000 mgr.y (mgr.25390) 506 : cluster 0 pgmap v358: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:45.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:44 smithi116 bash[65980]: audit 2023-07-10T17:18:44.521813+0000 mon.c (mon.1) 38 : audit 0 from='client.45073 172.21.15.116:0/2189054938' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:18:45.276 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2023-07-10T17:18:47.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:46 smithi192 bash[50362]: cluster 2023-07-10T17:18:45.349919+0000 mgr.y (mgr.25390) 507 : cluster 0 pgmap v359: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:47.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:46 smithi116 bash[63900]: cluster 2023-07-10T17:18:45.349919+0000 mgr.y (mgr.25390) 507 : cluster 0 pgmap v359: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:47.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:46 smithi116 bash[65980]: cluster 2023-07-10T17:18:45.349919+0000 mgr.y (mgr.25390) 507 : cluster 0 pgmap v359: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:48.384 INFO:teuthology.orchestra.run.smithi116.stdout:wait for servicemap items w/ changing names to refresh 2023-07-10T17:18:49.145 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:48 smithi192 bash[50362]: cluster 2023-07-10T17:18:47.351473+0000 mgr.y (mgr.25390) 508 : cluster 0 pgmap v360: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:49.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:48 smithi116 bash[63900]: cluster 2023-07-10T17:18:47.351473+0000 mgr.y (mgr.25390) 508 : cluster 0 pgmap v360: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:49.146 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:48 smithi116 bash[65980]: cluster 2023-07-10T17:18:47.351473+0000 mgr.y (mgr.25390) 508 : cluster 0 pgmap v360: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:49.147 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'sleep 60' 2023-07-10T17:18:51.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:50 smithi192 bash[50362]: cluster 2023-07-10T17:18:49.352309+0000 mgr.y (mgr.25390) 509 : cluster 0 pgmap v361: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:51.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:50 smithi116 bash[63900]: cluster 2023-07-10T17:18:49.352309+0000 mgr.y (mgr.25390) 509 : cluster 0 pgmap v361: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:51.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:50 smithi116 bash[65980]: cluster 2023-07-10T17:18:49.352309+0000 mgr.y (mgr.25390) 509 : cluster 0 pgmap v361: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:52.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:51 smithi192 bash[50362]: audit 2023-07-10T17:18:51.301603+0000 mgr.y (mgr.25390) 510 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:52.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:51 smithi116 bash[63900]: audit 2023-07-10T17:18:51.301603+0000 mgr.y (mgr.25390) 510 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:52.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:51 smithi116 bash[65980]: audit 2023-07-10T17:18:51.301603+0000 mgr.y (mgr.25390) 510 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:18:53.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:52 smithi192 bash[50362]: cluster 2023-07-10T17:18:51.353554+0000 mgr.y (mgr.25390) 511 : cluster 0 pgmap v362: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:53.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:52 smithi116 bash[63900]: cluster 2023-07-10T17:18:51.353554+0000 mgr.y (mgr.25390) 511 : cluster 0 pgmap v362: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:53.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:52 smithi116 bash[65980]: cluster 2023-07-10T17:18:51.353554+0000 mgr.y (mgr.25390) 511 : cluster 0 pgmap v362: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:55.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:54 smithi192 bash[50362]: cluster 2023-07-10T17:18:53.354692+0000 mgr.y (mgr.25390) 512 : cluster 0 pgmap v363: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:55.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:54 smithi116 bash[63900]: cluster 2023-07-10T17:18:53.354692+0000 mgr.y (mgr.25390) 512 : cluster 0 pgmap v363: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:55.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:54 smithi116 bash[65980]: cluster 2023-07-10T17:18:53.354692+0000 mgr.y (mgr.25390) 512 : cluster 0 pgmap v363: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:57.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:56 smithi192 bash[50362]: cluster 2023-07-10T17:18:55.355762+0000 mgr.y (mgr.25390) 513 : cluster 0 pgmap v364: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:57.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:56 smithi116 bash[63900]: cluster 2023-07-10T17:18:55.355762+0000 mgr.y (mgr.25390) 513 : cluster 0 pgmap v364: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:57.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:56 smithi116 bash[65980]: cluster 2023-07-10T17:18:55.355762+0000 mgr.y (mgr.25390) 513 : cluster 0 pgmap v364: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:18:59.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:18:58 smithi192 bash[50362]: cluster 2023-07-10T17:18:57.357228+0000 mgr.y (mgr.25390) 514 : cluster 0 pgmap v365: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:59.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:18:58 smithi116 bash[63900]: cluster 2023-07-10T17:18:57.357228+0000 mgr.y (mgr.25390) 514 : cluster 0 pgmap v365: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:18:59.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:18:58 smithi116 bash[65980]: cluster 2023-07-10T17:18:57.357228+0000 mgr.y (mgr.25390) 514 : cluster 0 pgmap v365: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:01.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:00 smithi192 bash[50362]: cluster 2023-07-10T17:18:59.358174+0000 mgr.y (mgr.25390) 515 : cluster 0 pgmap v366: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:01.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:00 smithi116 bash[63900]: cluster 2023-07-10T17:18:59.358174+0000 mgr.y (mgr.25390) 515 : cluster 0 pgmap v366: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:01.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:00 smithi116 bash[65980]: cluster 2023-07-10T17:18:59.358174+0000 mgr.y (mgr.25390) 515 : cluster 0 pgmap v366: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:03.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:02 smithi192 bash[50362]: audit 2023-07-10T17:19:01.312848+0000 mgr.y (mgr.25390) 516 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:03.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:02 smithi192 bash[50362]: cluster 2023-07-10T17:19:01.359459+0000 mgr.y (mgr.25390) 517 : cluster 0 pgmap v367: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:03.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:02 smithi116 bash[63900]: audit 2023-07-10T17:19:01.312848+0000 mgr.y (mgr.25390) 516 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:03.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:02 smithi116 bash[63900]: cluster 2023-07-10T17:19:01.359459+0000 mgr.y (mgr.25390) 517 : cluster 0 pgmap v367: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:03.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:02 smithi116 bash[65980]: audit 2023-07-10T17:19:01.312848+0000 mgr.y (mgr.25390) 516 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:03.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:02 smithi116 bash[65980]: cluster 2023-07-10T17:19:01.359459+0000 mgr.y (mgr.25390) 517 : cluster 0 pgmap v367: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:05.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:04 smithi192 bash[50362]: cluster 2023-07-10T17:19:03.360556+0000 mgr.y (mgr.25390) 518 : cluster 0 pgmap v368: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:05.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:04 smithi116 bash[63900]: cluster 2023-07-10T17:19:03.360556+0000 mgr.y (mgr.25390) 518 : cluster 0 pgmap v368: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:05.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:04 smithi116 bash[65980]: cluster 2023-07-10T17:19:03.360556+0000 mgr.y (mgr.25390) 518 : cluster 0 pgmap v368: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:07.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:06 smithi192 bash[50362]: cluster 2023-07-10T17:19:05.361626+0000 mgr.y (mgr.25390) 519 : cluster 0 pgmap v369: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:07.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:06 smithi116 bash[63900]: cluster 2023-07-10T17:19:05.361626+0000 mgr.y (mgr.25390) 519 : cluster 0 pgmap v369: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:07.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:06 smithi116 bash[65980]: cluster 2023-07-10T17:19:05.361626+0000 mgr.y (mgr.25390) 519 : cluster 0 pgmap v369: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:09.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:08 smithi192 bash[50362]: cluster 2023-07-10T17:19:07.363087+0000 mgr.y (mgr.25390) 520 : cluster 0 pgmap v370: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:09.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:08 smithi116 bash[63900]: cluster 2023-07-10T17:19:07.363087+0000 mgr.y (mgr.25390) 520 : cluster 0 pgmap v370: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:09.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:08 smithi116 bash[65980]: cluster 2023-07-10T17:19:07.363087+0000 mgr.y (mgr.25390) 520 : cluster 0 pgmap v370: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:11.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:10 smithi192 bash[50362]: cluster 2023-07-10T17:19:09.364050+0000 mgr.y (mgr.25390) 521 : cluster 0 pgmap v371: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:11.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:10 smithi116 bash[63900]: cluster 2023-07-10T17:19:09.364050+0000 mgr.y (mgr.25390) 521 : cluster 0 pgmap v371: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:11.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:10 smithi116 bash[65980]: cluster 2023-07-10T17:19:09.364050+0000 mgr.y (mgr.25390) 521 : cluster 0 pgmap v371: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:13.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:12 smithi116 bash[63900]: audit 2023-07-10T17:19:11.324148+0000 mgr.y (mgr.25390) 522 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:13.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:12 smithi116 bash[63900]: cluster 2023-07-10T17:19:11.365404+0000 mgr.y (mgr.25390) 523 : cluster 0 pgmap v372: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:13.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:12 smithi116 bash[65980]: audit 2023-07-10T17:19:11.324148+0000 mgr.y (mgr.25390) 522 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:13.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:12 smithi116 bash[65980]: cluster 2023-07-10T17:19:11.365404+0000 mgr.y (mgr.25390) 523 : cluster 0 pgmap v372: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:13.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:12 smithi192 bash[50362]: audit 2023-07-10T17:19:11.324148+0000 mgr.y (mgr.25390) 522 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:13.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:12 smithi192 bash[50362]: cluster 2023-07-10T17:19:11.365404+0000 mgr.y (mgr.25390) 523 : cluster 0 pgmap v372: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:15.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:14 smithi192 bash[50362]: cluster 2023-07-10T17:19:13.366668+0000 mgr.y (mgr.25390) 524 : cluster 0 pgmap v373: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:15.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:14 smithi116 bash[65980]: cluster 2023-07-10T17:19:13.366668+0000 mgr.y (mgr.25390) 524 : cluster 0 pgmap v373: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:15.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:14 smithi116 bash[63900]: cluster 2023-07-10T17:19:13.366668+0000 mgr.y (mgr.25390) 524 : cluster 0 pgmap v373: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:17.394 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:16 smithi192 bash[50362]: cluster 2023-07-10T17:19:15.367819+0000 mgr.y (mgr.25390) 525 : cluster 0 pgmap v374: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:17.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:16 smithi116 bash[63900]: cluster 2023-07-10T17:19:15.367819+0000 mgr.y (mgr.25390) 525 : cluster 0 pgmap v374: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:17.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:16 smithi116 bash[65980]: cluster 2023-07-10T17:19:15.367819+0000 mgr.y (mgr.25390) 525 : cluster 0 pgmap v374: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:19.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:18 smithi192 bash[50362]: cluster 2023-07-10T17:19:17.369411+0000 mgr.y (mgr.25390) 526 : cluster 0 pgmap v375: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:19.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:18 smithi116 bash[63900]: cluster 2023-07-10T17:19:17.369411+0000 mgr.y (mgr.25390) 526 : cluster 0 pgmap v375: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:19.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:18 smithi116 bash[65980]: cluster 2023-07-10T17:19:17.369411+0000 mgr.y (mgr.25390) 526 : cluster 0 pgmap v375: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:21.339 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:20 smithi116 bash[63900]: cluster 2023-07-10T17:19:19.370283+0000 mgr.y (mgr.25390) 527 : cluster 0 pgmap v376: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:21.339 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:20 smithi116 bash[65980]: cluster 2023-07-10T17:19:19.370283+0000 mgr.y (mgr.25390) 527 : cluster 0 pgmap v376: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:21.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:20 smithi192 bash[50362]: cluster 2023-07-10T17:19:19.370283+0000 mgr.y (mgr.25390) 527 : cluster 0 pgmap v376: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:22.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:21 smithi192 bash[50362]: audit 2023-07-10T17:19:21.461778+0000 mon.a (mon.0) 1155 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:19:22.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:21 smithi192 bash[50362]: audit 2023-07-10T17:19:21.463665+0000 mon.a (mon.0) 1156 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:19:22.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:21 smithi192 bash[50362]: audit 2023-07-10T17:19:21.474074+0000 mon.a (mon.0) 1157 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:19:22.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:21 smithi116 bash[65980]: audit 2023-07-10T17:19:21.461778+0000 mon.a (mon.0) 1155 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:19:22.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:21 smithi116 bash[65980]: audit 2023-07-10T17:19:21.463665+0000 mon.a (mon.0) 1156 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:19:22.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:21 smithi116 bash[65980]: audit 2023-07-10T17:19:21.474074+0000 mon.a (mon.0) 1157 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:19:22.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:21 smithi116 bash[63900]: audit 2023-07-10T17:19:21.461778+0000 mon.a (mon.0) 1155 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:19:22.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:21 smithi116 bash[63900]: audit 2023-07-10T17:19:21.463665+0000 mon.a (mon.0) 1156 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:19:22.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:21 smithi116 bash[63900]: audit 2023-07-10T17:19:21.474074+0000 mon.a (mon.0) 1157 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:19:23.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:22 smithi192 bash[50362]: audit 2023-07-10T17:19:21.334366+0000 mgr.y (mgr.25390) 528 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:23.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:22 smithi192 bash[50362]: cluster 2023-07-10T17:19:21.371344+0000 mgr.y (mgr.25390) 529 : cluster 0 pgmap v377: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:23.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:22 smithi192 bash[50362]: cephadm 2023-07-10T17:19:21.465295+0000 mgr.y (mgr.25390) 530 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:19:23.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:22 smithi116 bash[65980]: audit 2023-07-10T17:19:21.334366+0000 mgr.y (mgr.25390) 528 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:23.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:22 smithi116 bash[65980]: cluster 2023-07-10T17:19:21.371344+0000 mgr.y (mgr.25390) 529 : cluster 0 pgmap v377: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:23.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:22 smithi116 bash[65980]: cephadm 2023-07-10T17:19:21.465295+0000 mgr.y (mgr.25390) 530 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:19:23.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:22 smithi116 bash[63900]: audit 2023-07-10T17:19:21.334366+0000 mgr.y (mgr.25390) 528 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:23.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:22 smithi116 bash[63900]: cluster 2023-07-10T17:19:21.371344+0000 mgr.y (mgr.25390) 529 : cluster 0 pgmap v377: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:23.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:22 smithi116 bash[63900]: cephadm 2023-07-10T17:19:21.465295+0000 mgr.y (mgr.25390) 530 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:19:25.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:24 smithi192 bash[50362]: cluster 2023-07-10T17:19:23.372410+0000 mgr.y (mgr.25390) 531 : cluster 0 pgmap v378: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:25.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:24 smithi116 bash[65980]: cluster 2023-07-10T17:19:23.372410+0000 mgr.y (mgr.25390) 531 : cluster 0 pgmap v378: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:25.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:24 smithi116 bash[63900]: cluster 2023-07-10T17:19:23.372410+0000 mgr.y (mgr.25390) 531 : cluster 0 pgmap v378: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:27.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:26 smithi192 bash[50362]: cluster 2023-07-10T17:19:25.373488+0000 mgr.y (mgr.25390) 532 : cluster 0 pgmap v379: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:27.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:26 smithi116 bash[65980]: cluster 2023-07-10T17:19:25.373488+0000 mgr.y (mgr.25390) 532 : cluster 0 pgmap v379: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:27.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:26 smithi116 bash[63900]: cluster 2023-07-10T17:19:25.373488+0000 mgr.y (mgr.25390) 532 : cluster 0 pgmap v379: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:29.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:28 smithi192 bash[50362]: cluster 2023-07-10T17:19:27.374980+0000 mgr.y (mgr.25390) 533 : cluster 0 pgmap v380: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:29.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:28 smithi116 bash[63900]: cluster 2023-07-10T17:19:27.374980+0000 mgr.y (mgr.25390) 533 : cluster 0 pgmap v380: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:29.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:28 smithi116 bash[65980]: cluster 2023-07-10T17:19:27.374980+0000 mgr.y (mgr.25390) 533 : cluster 0 pgmap v380: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:31.350 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:30 smithi116 bash[63900]: cluster 2023-07-10T17:19:29.375869+0000 mgr.y (mgr.25390) 534 : cluster 0 pgmap v381: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:31.351 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:30 smithi116 bash[65980]: cluster 2023-07-10T17:19:29.375869+0000 mgr.y (mgr.25390) 534 : cluster 0 pgmap v381: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:31.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:30 smithi192 bash[50362]: cluster 2023-07-10T17:19:29.375869+0000 mgr.y (mgr.25390) 534 : cluster 0 pgmap v381: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:33.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:32 smithi192 bash[50362]: audit 2023-07-10T17:19:31.345449+0000 mgr.y (mgr.25390) 535 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:33.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:32 smithi192 bash[50362]: cluster 2023-07-10T17:19:31.377130+0000 mgr.y (mgr.25390) 536 : cluster 0 pgmap v382: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:33.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:32 smithi116 bash[63900]: audit 2023-07-10T17:19:31.345449+0000 mgr.y (mgr.25390) 535 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:33.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:32 smithi116 bash[63900]: cluster 2023-07-10T17:19:31.377130+0000 mgr.y (mgr.25390) 536 : cluster 0 pgmap v382: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:33.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:32 smithi116 bash[65980]: audit 2023-07-10T17:19:31.345449+0000 mgr.y (mgr.25390) 535 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:33.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:32 smithi116 bash[65980]: cluster 2023-07-10T17:19:31.377130+0000 mgr.y (mgr.25390) 536 : cluster 0 pgmap v382: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:35.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:34 smithi192 bash[50362]: cluster 2023-07-10T17:19:33.378484+0000 mgr.y (mgr.25390) 537 : cluster 0 pgmap v383: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:35.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:34 smithi116 bash[63900]: cluster 2023-07-10T17:19:33.378484+0000 mgr.y (mgr.25390) 537 : cluster 0 pgmap v383: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:35.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:34 smithi116 bash[65980]: cluster 2023-07-10T17:19:33.378484+0000 mgr.y (mgr.25390) 537 : cluster 0 pgmap v383: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:37.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:36 smithi192 bash[50362]: cluster 2023-07-10T17:19:35.379589+0000 mgr.y (mgr.25390) 538 : cluster 0 pgmap v384: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:37.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:36 smithi116 bash[63900]: cluster 2023-07-10T17:19:35.379589+0000 mgr.y (mgr.25390) 538 : cluster 0 pgmap v384: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:37.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:36 smithi116 bash[65980]: cluster 2023-07-10T17:19:35.379589+0000 mgr.y (mgr.25390) 538 : cluster 0 pgmap v384: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:39.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:38 smithi192 bash[50362]: cluster 2023-07-10T17:19:37.380991+0000 mgr.y (mgr.25390) 539 : cluster 0 pgmap v385: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:39.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:38 smithi116 bash[63900]: cluster 2023-07-10T17:19:37.380991+0000 mgr.y (mgr.25390) 539 : cluster 0 pgmap v385: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:39.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:38 smithi116 bash[65980]: cluster 2023-07-10T17:19:37.380991+0000 mgr.y (mgr.25390) 539 : cluster 0 pgmap v385: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:41.361 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:40 smithi116 bash[63900]: cluster 2023-07-10T17:19:39.381909+0000 mgr.y (mgr.25390) 540 : cluster 0 pgmap v386: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:41.362 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:40 smithi116 bash[65980]: cluster 2023-07-10T17:19:39.381909+0000 mgr.y (mgr.25390) 540 : cluster 0 pgmap v386: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:41.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:40 smithi192 bash[50362]: cluster 2023-07-10T17:19:39.381909+0000 mgr.y (mgr.25390) 540 : cluster 0 pgmap v386: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:43.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:42 smithi192 bash[50362]: audit 2023-07-10T17:19:41.356727+0000 mgr.y (mgr.25390) 541 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:43.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:42 smithi192 bash[50362]: cluster 2023-07-10T17:19:41.383161+0000 mgr.y (mgr.25390) 542 : cluster 0 pgmap v387: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:43.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:42 smithi116 bash[63900]: audit 2023-07-10T17:19:41.356727+0000 mgr.y (mgr.25390) 541 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:43.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:42 smithi116 bash[63900]: cluster 2023-07-10T17:19:41.383161+0000 mgr.y (mgr.25390) 542 : cluster 0 pgmap v387: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:43.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:42 smithi116 bash[65980]: audit 2023-07-10T17:19:41.356727+0000 mgr.y (mgr.25390) 541 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:43.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:42 smithi116 bash[65980]: cluster 2023-07-10T17:19:41.383161+0000 mgr.y (mgr.25390) 542 : cluster 0 pgmap v387: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:45.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:44 smithi192 bash[50362]: cluster 2023-07-10T17:19:43.384300+0000 mgr.y (mgr.25390) 543 : cluster 0 pgmap v388: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:45.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:44 smithi116 bash[63900]: cluster 2023-07-10T17:19:43.384300+0000 mgr.y (mgr.25390) 543 : cluster 0 pgmap v388: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:45.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:44 smithi116 bash[65980]: cluster 2023-07-10T17:19:43.384300+0000 mgr.y (mgr.25390) 543 : cluster 0 pgmap v388: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:47.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:46 smithi192 bash[50362]: cluster 2023-07-10T17:19:45.385403+0000 mgr.y (mgr.25390) 544 : cluster 0 pgmap v389: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:47.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:46 smithi116 bash[63900]: cluster 2023-07-10T17:19:45.385403+0000 mgr.y (mgr.25390) 544 : cluster 0 pgmap v389: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:47.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:46 smithi116 bash[65980]: cluster 2023-07-10T17:19:45.385403+0000 mgr.y (mgr.25390) 544 : cluster 0 pgmap v389: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:49.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:48 smithi192 bash[50362]: cluster 2023-07-10T17:19:47.386879+0000 mgr.y (mgr.25390) 545 : cluster 0 pgmap v390: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:49.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:48 smithi116 bash[63900]: cluster 2023-07-10T17:19:47.386879+0000 mgr.y (mgr.25390) 545 : cluster 0 pgmap v390: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:49.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:48 smithi116 bash[65980]: cluster 2023-07-10T17:19:47.386879+0000 mgr.y (mgr.25390) 545 : cluster 0 pgmap v390: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:51.376 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:50 smithi116 bash[63900]: cluster 2023-07-10T17:19:49.387868+0000 mgr.y (mgr.25390) 546 : cluster 0 pgmap v391: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:51.377 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:50 smithi116 bash[65980]: cluster 2023-07-10T17:19:49.387868+0000 mgr.y (mgr.25390) 546 : cluster 0 pgmap v391: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:51.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:50 smithi192 bash[50362]: cluster 2023-07-10T17:19:49.387868+0000 mgr.y (mgr.25390) 546 : cluster 0 pgmap v391: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:53.116 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph orch ps' 2023-07-10T17:19:53.305 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:52 smithi116 bash[63900]: audit 2023-07-10T17:19:51.367469+0000 mgr.y (mgr.25390) 547 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:53.305 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:52 smithi116 bash[63900]: cluster 2023-07-10T17:19:51.389167+0000 mgr.y (mgr.25390) 548 : cluster 0 pgmap v392: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:53.306 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:52 smithi116 bash[65980]: audit 2023-07-10T17:19:51.367469+0000 mgr.y (mgr.25390) 547 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:53.306 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:52 smithi116 bash[65980]: cluster 2023-07-10T17:19:51.389167+0000 mgr.y (mgr.25390) 548 : cluster 0 pgmap v392: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:53.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:52 smithi192 bash[50362]: audit 2023-07-10T17:19:51.367469+0000 mgr.y (mgr.25390) 547 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:19:53.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:52 smithi192 bash[50362]: cluster 2023-07-10T17:19:51.389167+0000 mgr.y (mgr.25390) 548 : cluster 0 pgmap v392: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:55.266 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:54 smithi116 bash[65980]: cluster 2023-07-10T17:19:53.390395+0000 mgr.y (mgr.25390) 549 : cluster 0 pgmap v393: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:55.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:54 smithi116 bash[63900]: cluster 2023-07-10T17:19:53.390395+0000 mgr.y (mgr.25390) 549 : cluster 0 pgmap v393: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:55.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:54 smithi192 bash[50362]: cluster 2023-07-10T17:19:53.390395+0000 mgr.y (mgr.25390) 549 : cluster 0 pgmap v393: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:56.750 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-10T17:19:56.750 INFO:teuthology.orchestra.run.smithi116.stdout:alertmanager.a smithi116 *:9093,9094 running (2m) 2m ago 19m 15.6M - 0.25.0 c8568f914cd2 5032fe494a7b 2023-07-10T17:19:56.751 INFO:teuthology.orchestra.run.smithi116.stdout:grafana.a smithi192 *:3000 running (115s) 109s ago 19m 25.1M - 9.4.7 2c41d148cca3 cb0b03556cc9 2023-07-10T17:19:56.751 INFO:teuthology.orchestra.run.smithi116.stdout:iscsi.foo.smithi116.fkthyl smithi116 running (3m) 2m ago 16m 48.0M - 3.6 17e8f0d79247 fbdecd8618b7 2023-07-10T17:19:56.751 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi192 *:8443,8765 running (10m) 109s ago 23m 431M - 18.1.2-30-g8507faa7 17e8f0d79247 4036d90533ec 2023-07-10T17:19:56.751 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi116 *:8443,8765 running (11m) 2m ago 25m 487M - 18.1.2-30-g8507faa7 17e8f0d79247 5c21c164d874 2023-07-10T17:19:56.751 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (10m) 2m ago 25m 84.1M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 ffe4994aed91 2023-07-10T17:19:56.751 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi192 running (9m) 109s ago 24m 62.5M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 74a082a2e795 2023-07-10T17:19:56.751 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi116 running (9m) 2m ago 24m 63.0M 2048M 18.1.2-30-g8507faa7 17e8f0d79247 a4e85815a0ef 2023-07-10T17:19:56.752 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.a smithi116 *:9100 running (3m) 2m ago 19m 9140k - 1.5.0 0da6a335fe13 64b482421ffb 2023-07-10T17:19:56.752 INFO:teuthology.orchestra.run.smithi116.stdout:node-exporter.b smithi192 *:9100 running (3m) 109s ago 19m 9744k - 1.5.0 0da6a335fe13 82b166142066 2023-07-10T17:19:56.756 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (8m) 2m ago 23m 63.7M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 0562f587fbca 2023-07-10T17:19:56.756 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi116 running (7m) 2m ago 22m 63.5M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 5b2a12a0bfc9 2023-07-10T17:19:56.757 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi116 running (7m) 2m ago 22m 56.1M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 88096bc0faa9 2023-07-10T17:19:56.757 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (6m) 2m ago 22m 61.2M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 61ae280c471b 2023-07-10T17:19:56.757 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi192 running (6m) 109s ago 21m 62.3M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 44d809c18118 2023-07-10T17:19:56.757 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi192 running (5m) 109s ago 21m 58.3M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 179270eb7044 2023-07-10T17:19:56.757 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi192 running (5m) 109s ago 20m 55.4M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 b680bfbe9a0c 2023-07-10T17:19:56.757 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi192 running (4m) 109s ago 20m 57.4M 4096M 18.1.2-30-g8507faa7 17e8f0d79247 bd3788803e68 2023-07-10T17:19:56.757 INFO:teuthology.orchestra.run.smithi116.stdout:prometheus.a smithi192 *:9095 running (2m) 109s ago 19m 37.0M - 2.43.0 a07b618ecd1d 48c15c1430f7 2023-07-10T17:19:56.758 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi116.stnxgn smithi116 *:8000 running (4m) 2m ago 17m 84.3M - 18.1.2-30-g8507faa7 17e8f0d79247 ef72a4e0ae29 2023-07-10T17:19:56.758 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.foo.smithi192.myrhsb smithi192 *:8000 running (4m) 109s ago 17m 84.7M - 18.1.2-30-g8507faa7 17e8f0d79247 59b0f0b41f44 2023-07-10T17:19:56.758 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi116.vpoihf smithi116 *:80 running (4m) 2m ago 17m 84.4M - 18.1.2-30-g8507faa7 17e8f0d79247 42f4f263b75e 2023-07-10T17:19:56.758 INFO:teuthology.orchestra.run.smithi116.stdout:rgw.smpl.smithi192.yqkmva smithi192 *:80 running (3m) 109s ago 17m 84.3M - 18.1.2-30-g8507faa7 17e8f0d79247 2a257a88c4e9 2023-07-10T17:19:57.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:56 smithi116 bash[63900]: cluster 2023-07-10T17:19:55.391474+0000 mgr.y (mgr.25390) 550 : cluster 0 pgmap v394: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:57.144 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:56 smithi116 bash[65980]: cluster 2023-07-10T17:19:55.391474+0000 mgr.y (mgr.25390) 550 : cluster 0 pgmap v394: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:57.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:56 smithi192 bash[50362]: cluster 2023-07-10T17:19:55.391474+0000 mgr.y (mgr.25390) 550 : cluster 0 pgmap v394: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:19:57.498 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph versions' 2023-07-10T17:19:58.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:58 smithi192 bash[50362]: audit 2023-07-10T17:19:56.719893+0000 mgr.y (mgr.25390) 551 : audit 0 from='client.35217 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:19:58.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:58 smithi116 bash[65980]: audit 2023-07-10T17:19:56.719893+0000 mgr.y (mgr.25390) 551 : audit 0 from='client.35217 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:19:58.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:58 smithi116 bash[63900]: audit 2023-07-10T17:19:56.719893+0000 mgr.y (mgr.25390) 551 : audit 0 from='client.35217 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:19:59.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:19:59 smithi116 bash[63900]: cluster 2023-07-10T17:19:57.392922+0000 mgr.y (mgr.25390) 552 : cluster 0 pgmap v395: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:59.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:19:59 smithi192 bash[50362]: cluster 2023-07-10T17:19:57.392922+0000 mgr.y (mgr.25390) 552 : cluster 0 pgmap v395: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:19:59.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:19:59 smithi116 bash[65980]: cluster 2023-07-10T17:19:57.392922+0000 mgr.y (mgr.25390) 552 : cluster 0 pgmap v395: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:20:00.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:00 smithi192 bash[50362]: cluster 2023-07-10T17:20:00.000147+0000 mon.a (mon.0) 1158 : cluster 1 overall HEALTH_OK 2023-07-10T17:20:00.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:00 smithi116 bash[63900]: cluster 2023-07-10T17:20:00.000147+0000 mon.a (mon.0) 1158 : cluster 1 overall HEALTH_OK 2023-07-10T17:20:00.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:00 smithi116 bash[65980]: cluster 2023-07-10T17:20:00.000147+0000 mon.a (mon.0) 1158 : cluster 1 overall HEALTH_OK 2023-07-10T17:20:01.214 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:20:01.215 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": { 2023-07-10T17:20:01.215 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 3 2023-07-10T17:20:01.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:20:01.215 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": { 2023-07-10T17:20:01.215 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 2 2023-07-10T17:20:01.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:20:01.216 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": { 2023-07-10T17:20:01.216 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 8 2023-07-10T17:20:01.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:20:01.216 INFO:teuthology.orchestra.run.smithi116.stdout: "rgw": { 2023-07-10T17:20:01.216 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 4 2023-07-10T17:20:01.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-07-10T17:20:01.217 INFO:teuthology.orchestra.run.smithi116.stdout: "overall": { 2023-07-10T17:20:01.217 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)": 17 2023-07-10T17:20:01.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-07-10T17:20:01.217 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:20:01.351 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:01 smithi116 bash[65980]: cluster 2023-07-10T17:19:59.393851+0000 mgr.y (mgr.25390) 553 : cluster 0 pgmap v396: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:01.351 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:01 smithi116 bash[63900]: cluster 2023-07-10T17:19:59.393851+0000 mgr.y (mgr.25390) 553 : cluster 0 pgmap v396: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:01.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:01 smithi192 bash[50362]: cluster 2023-07-10T17:19:59.393851+0000 mgr.y (mgr.25390) 553 : cluster 0 pgmap v396: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:01.975 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph orch upgrade status' 2023-07-10T17:20:02.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:02 smithi192 bash[50362]: audit 2023-07-10T17:20:01.209424+0000 mon.a (mon.0) 1159 : audit 0 from='client.35223 172.21.15.116:0/539483733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:20:02.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:02 smithi116 bash[63900]: audit 2023-07-10T17:20:01.209424+0000 mon.a (mon.0) 1159 : audit 0 from='client.35223 172.21.15.116:0/539483733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:20:02.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:02 smithi116 bash[65980]: audit 2023-07-10T17:20:01.209424+0000 mon.a (mon.0) 1159 : audit 0 from='client.35223 172.21.15.116:0/539483733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:20:03.312 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:03 smithi116 bash[63900]: audit 2023-07-10T17:20:01.377572+0000 mgr.y (mgr.25390) 554 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:20:03.313 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:03 smithi116 bash[63900]: cluster 2023-07-10T17:20:01.395082+0000 mgr.y (mgr.25390) 555 : cluster 0 pgmap v397: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:20:03.313 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:03 smithi116 bash[65980]: audit 2023-07-10T17:20:01.377572+0000 mgr.y (mgr.25390) 554 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:20:03.313 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:03 smithi116 bash[65980]: cluster 2023-07-10T17:20:01.395082+0000 mgr.y (mgr.25390) 555 : cluster 0 pgmap v397: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:20:03.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:03 smithi192 bash[50362]: audit 2023-07-10T17:20:01.377572+0000 mgr.y (mgr.25390) 554 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:20:03.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:03 smithi192 bash[50362]: cluster 2023-07-10T17:20:01.395082+0000 mgr.y (mgr.25390) 555 : cluster 0 pgmap v397: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:20:05.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:05 smithi192 bash[50362]: cluster 2023-07-10T17:20:03.396240+0000 mgr.y (mgr.25390) 556 : cluster 0 pgmap v398: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:05.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:05 smithi116 bash[63900]: cluster 2023-07-10T17:20:03.396240+0000 mgr.y (mgr.25390) 556 : cluster 0 pgmap v398: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:05.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:05 smithi116 bash[65980]: cluster 2023-07-10T17:20:03.396240+0000 mgr.y (mgr.25390) 556 : cluster 0 pgmap v398: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:05.493 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-07-10T17:20:05.496 INFO:teuthology.orchestra.run.smithi116.stdout: "target_image": null, 2023-07-10T17:20:05.497 INFO:teuthology.orchestra.run.smithi116.stdout: "in_progress": false, 2023-07-10T17:20:05.497 INFO:teuthology.orchestra.run.smithi116.stdout: "which": "", 2023-07-10T17:20:05.497 INFO:teuthology.orchestra.run.smithi116.stdout: "services_complete": [], 2023-07-10T17:20:05.497 INFO:teuthology.orchestra.run.smithi116.stdout: "progress": null, 2023-07-10T17:20:05.497 INFO:teuthology.orchestra.run.smithi116.stdout: "message": "", 2023-07-10T17:20:05.497 INFO:teuthology.orchestra.run.smithi116.stdout: "is_paused": false 2023-07-10T17:20:05.497 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-07-10T17:20:06.274 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph health detail' 2023-07-10T17:20:07.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:07 smithi192 bash[50362]: cluster 2023-07-10T17:20:05.397268+0000 mgr.y (mgr.25390) 557 : cluster 0 pgmap v399: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:07.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:07 smithi192 bash[50362]: audit 2023-07-10T17:20:05.488894+0000 mgr.y (mgr.25390) 558 : audit 0 from='client.45091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:20:07.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:07 smithi116 bash[63900]: cluster 2023-07-10T17:20:05.397268+0000 mgr.y (mgr.25390) 557 : cluster 0 pgmap v399: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:07.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:07 smithi116 bash[63900]: audit 2023-07-10T17:20:05.488894+0000 mgr.y (mgr.25390) 558 : audit 0 from='client.45091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:20:07.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:07 smithi116 bash[65980]: cluster 2023-07-10T17:20:05.397268+0000 mgr.y (mgr.25390) 557 : cluster 0 pgmap v399: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:07.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:07 smithi116 bash[65980]: audit 2023-07-10T17:20:05.488894+0000 mgr.y (mgr.25390) 558 : audit 0 from='client.45091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:20:09.314 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:09 smithi116 bash[63900]: cluster 2023-07-10T17:20:07.398791+0000 mgr.y (mgr.25390) 559 : cluster 0 pgmap v400: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:20:09.315 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:09 smithi116 bash[65980]: cluster 2023-07-10T17:20:07.398791+0000 mgr.y (mgr.25390) 559 : cluster 0 pgmap v400: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:20:09.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:09 smithi192 bash[50362]: cluster 2023-07-10T17:20:07.398791+0000 mgr.y (mgr.25390) 559 : cluster 0 pgmap v400: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:20:09.812 INFO:teuthology.orchestra.run.smithi116.stdout:HEALTH_OK 2023-07-10T17:20:10.148 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:10 smithi116 bash[63900]: audit 2023-07-10T17:20:09.806900+0000 mon.a (mon.0) 1160 : audit 0 from='client.35235 172.21.15.116:0/1934536852' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:20:10.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:10 smithi116 bash[65980]: audit 2023-07-10T17:20:09.806900+0000 mon.a (mon.0) 1160 : audit 0 from='client.35235 172.21.15.116:0/1934536852' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:20:10.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:10 smithi192 bash[50362]: audit 2023-07-10T17:20:09.806900+0000 mon.a (mon.0) 1160 : audit 0 from='client.35235 172.21.15.116:0/1934536852' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-07-10T17:20:10.598 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2023-07-10T17:20:11.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:11 smithi192 bash[50362]: cluster 2023-07-10T17:20:09.399677+0000 mgr.y (mgr.25390) 560 : cluster 0 pgmap v401: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:11.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:11 smithi116 bash[63900]: cluster 2023-07-10T17:20:09.399677+0000 mgr.y (mgr.25390) 560 : cluster 0 pgmap v401: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:11.393 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:11 smithi116 bash[65980]: cluster 2023-07-10T17:20:09.399677+0000 mgr.y (mgr.25390) 560 : cluster 0 pgmap v401: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:13.078 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:13 smithi116 bash[63900]: audit 2023-07-10T17:20:11.387800+0000 mgr.y (mgr.25390) 561 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:20:13.078 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:13 smithi116 bash[63900]: cluster 2023-07-10T17:20:11.400956+0000 mgr.y (mgr.25390) 562 : cluster 0 pgmap v402: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:20:13.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:13 smithi192 bash[50362]: audit 2023-07-10T17:20:11.387800+0000 mgr.y (mgr.25390) 561 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:20:13.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:13 smithi192 bash[50362]: cluster 2023-07-10T17:20:11.400956+0000 mgr.y (mgr.25390) 562 : cluster 0 pgmap v402: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:20:13.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:13 smithi116 bash[65980]: audit 2023-07-10T17:20:11.387800+0000 mgr.y (mgr.25390) 561 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:20:13.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:13 smithi116 bash[65980]: cluster 2023-07-10T17:20:11.400956+0000 mgr.y (mgr.25390) 562 : cluster 0 pgmap v402: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:20:14.363 INFO:teuthology.orchestra.run.smithi116.stdout:true 2023-07-10T17:20:15.077 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2023-07-10T17:20:15.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:15 smithi192 bash[50362]: cluster 2023-07-10T17:20:13.402127+0000 mgr.y (mgr.25390) 563 : cluster 0 pgmap v403: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:15.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:15 smithi192 bash[50362]: audit 2023-07-10T17:20:14.342069+0000 mon.a (mon.0) 1161 : audit 0 from='client.45103 172.21.15.116:0/6572558' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:20:15.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:15 smithi116 bash[65980]: cluster 2023-07-10T17:20:13.402127+0000 mgr.y (mgr.25390) 563 : cluster 0 pgmap v403: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:15.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:15 smithi116 bash[65980]: audit 2023-07-10T17:20:14.342069+0000 mon.a (mon.0) 1161 : audit 0 from='client.45103 172.21.15.116:0/6572558' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:20:15.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:15 smithi116 bash[63900]: cluster 2023-07-10T17:20:13.402127+0000 mgr.y (mgr.25390) 563 : cluster 0 pgmap v403: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:15.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:15 smithi116 bash[63900]: audit 2023-07-10T17:20:14.342069+0000 mon.a (mon.0) 1161 : audit 0 from='client.45103 172.21.15.116:0/6572558' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:20:17.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:17 smithi192 bash[50362]: cluster 2023-07-10T17:20:15.403156+0000 mgr.y (mgr.25390) 564 : cluster 0 pgmap v404: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:17.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:17 smithi116 bash[63900]: cluster 2023-07-10T17:20:15.403156+0000 mgr.y (mgr.25390) 564 : cluster 0 pgmap v404: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:17.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:17 smithi116 bash[65980]: cluster 2023-07-10T17:20:15.403156+0000 mgr.y (mgr.25390) 564 : cluster 0 pgmap v404: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:18.822 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph version 18.1.2-30-g8507faa7 (8507faa7ce284e46218d481e3ee9c411ac3dc32c) reef (rc)" 2023-07-10T17:20:19.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:19 smithi116 bash[63900]: cluster 2023-07-10T17:20:17.404636+0000 mgr.y (mgr.25390) 565 : cluster 0 pgmap v405: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:20:19.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:19 smithi116 bash[63900]: audit 2023-07-10T17:20:18.802984+0000 mon.a (mon.0) 1162 : audit 0 from='client.35247 172.21.15.116:0/3412667055' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:20:19.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:19 smithi116 bash[65980]: cluster 2023-07-10T17:20:17.404636+0000 mgr.y (mgr.25390) 565 : cluster 0 pgmap v405: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:20:19.145 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:19 smithi116 bash[65980]: audit 2023-07-10T17:20:18.802984+0000 mon.a (mon.0) 1162 : audit 0 from='client.35247 172.21.15.116:0/3412667055' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:20:19.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:19 smithi192 bash[50362]: cluster 2023-07-10T17:20:17.404636+0000 mgr.y (mgr.25390) 565 : cluster 0 pgmap v405: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:20:19.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:19 smithi192 bash[50362]: audit 2023-07-10T17:20:18.802984+0000 mon.a (mon.0) 1162 : audit 0 from='client.35247 172.21.15.116:0/3412667055' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-07-10T17:20:19.588 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -e sha1=8507faa7ce284e46218d481e3ee9c411ac3dc32c -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2023-07-10T17:20:21.139 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:21 smithi116 bash[63900]: cluster 2023-07-10T17:20:19.405604+0000 mgr.y (mgr.25390) 566 : cluster 0 pgmap v406: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:21.139 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:21 smithi116 bash[65980]: cluster 2023-07-10T17:20:19.405604+0000 mgr.y (mgr.25390) 566 : cluster 0 pgmap v406: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:21.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:21 smithi192 bash[50362]: cluster 2023-07-10T17:20:19.405604+0000 mgr.y (mgr.25390) 566 : cluster 0 pgmap v406: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:23.150 INFO:teuthology.orchestra.run.smithi116.stdout:osd 8 2m ago - 2023-07-10T17:20:23.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:23 smithi192 bash[50362]: audit 2023-07-10T17:20:21.395755+0000 mgr.y (mgr.25390) 567 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:20:23.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:23 smithi192 bash[50362]: cluster 2023-07-10T17:20:21.406846+0000 mgr.y (mgr.25390) 568 : cluster 0 pgmap v407: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:20:23.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:23 smithi192 bash[50362]: audit 2023-07-10T17:20:22.095848+0000 mon.a (mon.0) 1163 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:20:23.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:23 smithi192 bash[50362]: audit 2023-07-10T17:20:22.097809+0000 mon.a (mon.0) 1164 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:20:23.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:23 smithi192 bash[50362]: cephadm 2023-07-10T17:20:22.099164+0000 mgr.y (mgr.25390) 569 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:20:23.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:23 smithi192 bash[50362]: audit 2023-07-10T17:20:22.106951+0000 mon.a (mon.0) 1165 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:20:23.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:23 smithi116 bash[63900]: audit 2023-07-10T17:20:21.395755+0000 mgr.y (mgr.25390) 567 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:20:23.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:23 smithi116 bash[63900]: cluster 2023-07-10T17:20:21.406846+0000 mgr.y (mgr.25390) 568 : cluster 0 pgmap v407: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:20:23.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:23 smithi116 bash[63900]: audit 2023-07-10T17:20:22.095848+0000 mon.a (mon.0) 1163 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:20:23.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:23 smithi116 bash[63900]: audit 2023-07-10T17:20:22.097809+0000 mon.a (mon.0) 1164 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:20:23.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:23 smithi116 bash[63900]: cephadm 2023-07-10T17:20:22.099164+0000 mgr.y (mgr.25390) 569 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:20:23.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:23 smithi116 bash[63900]: audit 2023-07-10T17:20:22.106951+0000 mon.a (mon.0) 1165 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:20:23.396 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:23 smithi116 bash[65980]: audit 2023-07-10T17:20:21.395755+0000 mgr.y (mgr.25390) 567 : audit 0 from='client.44728 -' entity='client.iscsi.foo.smithi116.fkthyl' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-10T17:20:23.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:23 smithi116 bash[65980]: cluster 2023-07-10T17:20:21.406846+0000 mgr.y (mgr.25390) 568 : cluster 0 pgmap v407: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-10T17:20:23.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:23 smithi116 bash[65980]: audit 2023-07-10T17:20:22.095848+0000 mon.a (mon.0) 1163 : audit 0 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-10T17:20:23.397 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:23 smithi116 bash[65980]: audit 2023-07-10T17:20:22.097809+0000 mon.a (mon.0) 1164 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-10T17:20:23.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:23 smithi116 bash[65980]: cephadm 2023-07-10T17:20:22.099164+0000 mgr.y (mgr.25390) 569 : cephadm 3 unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2023-07-10T17:20:23.398 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:23 smithi116 bash[65980]: audit 2023-07-10T17:20:22.106951+0000 mon.a (mon.0) 1165 : audit 1 from='mgr.25390 172.21.15.116:0/3022269787' entity='mgr.y' 2023-07-10T17:20:23.905 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-10T17:20:23.918 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2023-07-10T17:20:23.918 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- bash -c 'ceph orch upgrade ls' 2023-07-10T17:20:24.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:24 smithi192 bash[50362]: audit 2023-07-10T17:20:23.119744+0000 mgr.y (mgr.25390) 570 : audit 0 from='client.45115 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:20:24.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:24 smithi116 bash[63900]: audit 2023-07-10T17:20:23.119744+0000 mgr.y (mgr.25390) 570 : audit 0 from='client.45115 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:20:24.395 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:24 smithi116 bash[65980]: audit 2023-07-10T17:20:23.119744+0000 mgr.y (mgr.25390) 570 : audit 0 from='client.45115 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-10T17:20:25.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:25 smithi192 bash[50362]: cluster 2023-07-10T17:20:23.408088+0000 mgr.y (mgr.25390) 571 : cluster 0 pgmap v408: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:25.392 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:25 smithi116 bash[65980]: cluster 2023-07-10T17:20:23.408088+0000 mgr.y (mgr.25390) 571 : cluster 0 pgmap v408: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:25.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:25 smithi116 bash[63900]: cluster 2023-07-10T17:20:23.408088+0000 mgr.y (mgr.25390) 571 : cluster 0 pgmap v408: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:27.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:27 smithi192 bash[50362]: cluster 2023-07-10T17:20:25.409058+0000 mgr.y (mgr.25390) 572 : cluster 0 pgmap v409: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:27.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[63900]: cluster 2023-07-10T17:20:25.409058+0000 mgr.y (mgr.25390) 572 : cluster 0 pgmap v409: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:27.394 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[65980]: cluster 2023-07-10T17:20:25.409058+0000 mgr.y (mgr.25390) 572 : cluster 0 pgmap v409: 161 pgs: 161 active+clean; 604 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-10T17:20:27.857 INFO:teuthology.orchestra.run.smithi116.stderr:Error EINVAL: Traceback (most recent call last): 2023-07-10T17:20:27.858 INFO:teuthology.orchestra.run.smithi116.stderr: File "/usr/share/ceph/mgr/mgr_module.py", line 1774, in _handle_command 2023-07-10T17:20:27.858 INFO:teuthology.orchestra.run.smithi116.stderr: return self.handle_command(inbuf, cmd) 2023-07-10T17:20:27.858 INFO:teuthology.orchestra.run.smithi116.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in handle_command 2023-07-10T17:20:27.859 INFO:teuthology.orchestra.run.smithi116.stderr: return dispatch[cmd['prefix']].call(self, cmd, inbuf) 2023-07-10T17:20:27.859 INFO:teuthology.orchestra.run.smithi116.stderr: File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call 2023-07-10T17:20:27.859 INFO:teuthology.orchestra.run.smithi116.stderr: return self.func(mgr, **kwargs) 2023-07-10T17:20:27.859 INFO:teuthology.orchestra.run.smithi116.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in 2023-07-10T17:20:27.859 INFO:teuthology.orchestra.run.smithi116.stderr: wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 2023-07-10T17:20:27.860 INFO:teuthology.orchestra.run.smithi116.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper 2023-07-10T17:20:27.860 INFO:teuthology.orchestra.run.smithi116.stderr: return func(*args, **kwargs) 2023-07-10T17:20:27.860 INFO:teuthology.orchestra.run.smithi116.stderr: File "/usr/share/ceph/mgr/orchestrator/module.py", line 1708, in _upgrade_ls 2023-07-10T17:20:27.860 INFO:teuthology.orchestra.run.smithi116.stderr: r = raise_if_exception(completion) 2023-07-10T17:20:27.861 INFO:teuthology.orchestra.run.smithi116.stderr: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 228, in raise_if_exception 2023-07-10T17:20:27.861 INFO:teuthology.orchestra.run.smithi116.stderr: raise e 2023-07-10T17:20:27.861 INFO:teuthology.orchestra.run.smithi116.stderr:requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: https://quay.io/v2/auth?service=quay.io&scope=repository%3Aceph%2Fceph%253Av18%3Apull 2023-07-10T17:20:27.861 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-07-10T17:20:28.145 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: debug 2023-07-10T17:20:27.775+0000 7f69993a4700 -1 log_channel(cephadm) log [ERR] : 400 Client Error: BAD REQUEST for url: https://quay.io/v2/auth?service=quay.io&scope=repository%3Aceph%2Fceph%253Av18%3Apull 2023-07-10T17:20:28.145 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: Traceback (most recent call last): 2023-07-10T17:20:28.145 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper 2023-07-10T17:20:28.146 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: return OrchResult(f(*args, **kwargs)) 2023-07-10T17:20:28.146 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: File "/usr/share/ceph/mgr/cephadm/module.py", line 3184, in upgrade_ls 2023-07-10T17:20:28.147 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: return self.upgrade.upgrade_ls(image, tags, show_all_versions) 2023-07-10T17:20:28.147 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 280, in upgrade_ls 2023-07-10T17:20:28.147 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: ls = reg.get_tags(bare_image) 2023-07-10T17:20:28.148 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: File "/usr/share/ceph/mgr/cephadm/registry.py", line 50, in get_tags 2023-07-10T17:20:28.149 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: token = self.get_token(r) 2023-07-10T17:20:28.149 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: File "/usr/share/ceph/mgr/cephadm/registry.py", line 20, in get_token 2023-07-10T17:20:28.149 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: r.raise_for_status() 2023-07-10T17:20:28.149 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: File "/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status 2023-07-10T17:20:28.150 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: raise HTTPError(http_error_msg, response=self) 2023-07-10T17:20:28.150 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: https://quay.io/v2/auth?service=quay.io&scope=repository%3Aceph%2Fceph%253Av18%3Apull 2023-07-10T17:20:28.151 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: debug 2023-07-10T17:20:27.847+0000 7f69993a4700 -1 mgr handle_command module 'orchestrator' command handler threw exception: 400 Client Error: BAD REQUEST for url: https://quay.io/v2/auth?service=quay.io&scope=repository%3Aceph%2Fceph%253Av18%3Apull 2023-07-10T17:20:28.151 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: debug 2023-07-10T17:20:27.847+0000 7f69993a4700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last): 2023-07-10T17:20:28.151 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: File "/usr/share/ceph/mgr/mgr_module.py", line 1774, in _handle_command 2023-07-10T17:20:28.151 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: return self.handle_command(inbuf, cmd) 2023-07-10T17:20:28.152 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in handle_command 2023-07-10T17:20:28.152 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: return dispatch[cmd['prefix']].call(self, cmd, inbuf) 2023-07-10T17:20:28.152 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call 2023-07-10T17:20:28.152 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: return self.func(mgr, **kwargs) 2023-07-10T17:20:28.152 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in 2023-07-10T17:20:28.152 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 2023-07-10T17:20:28.153 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper 2023-07-10T17:20:28.153 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: return func(*args, **kwargs) 2023-07-10T17:20:28.153 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: File "/usr/share/ceph/mgr/orchestrator/module.py", line 1708, in _upgrade_ls 2023-07-10T17:20:28.153 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: r = raise_if_exception(completion) 2023-07-10T17:20:28.153 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 228, in raise_if_exception 2023-07-10T17:20:28.153 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: raise e 2023-07-10T17:20:28.153 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:27 smithi116 bash[59394]: requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: https://quay.io/v2/auth?service=quay.io&scope=repository%3Aceph%2Fceph%253Av18%3Apull 2023-07-10T17:20:28.654 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-07-10T17:20:28.655 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_8507faa7ce284e46218d481e3ee9c411ac3dc32c/qa/tasks/cephadm.py", line 1230, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_8507faa7ce284e46218d481e3ee9c411ac3dc32c/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- bash -c 'ceph orch upgrade ls'" 2023-07-10T17:20:28.740 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c0ea79b85234479bb1a232c51a6d93ad Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_8507faa7ce284e46218d481e3ee9c411ac3dc32c/qa/tasks/cephadm.py", line 1230, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_8507faa7ce284e46218d481e3ee9c411ac3dc32c/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- bash -c 'ceph orch upgrade ls'" 2023-07-10T17:20:28.743 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-10T17:20:28.769 INFO:tasks.cephadm:Teardown begin 2023-07-10T17:20:28.771 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-10T17:20:28.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-10T17:20:28.822 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-10T17:20:28.823 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-10T17:20:28.857 DEBUG:teuthology.orchestra.run.smithi192:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-10T17:20:28.864 INFO:tasks.cephadm:Stopping all daemons... 2023-07-10T17:20:28.865 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-10T17:20:28.866 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.a 2023-07-10T17:20:29.100 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:28 smithi116 systemd[1]: Stopping Ceph mon.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:20:29.395 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:29 smithi116 bash[63900]: debug 2023-07-10T17:20:29.095+0000 7f0584509700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-07-10T17:20:29.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Jul 10 17:20:29 smithi116 bash[63900]: debug 2023-07-10T17:20:29.095+0000 7f0584509700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2023-07-10T17:20:29.990 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.a.service' 2023-07-10T17:20:30.033 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-10T17:20:30.035 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-10T17:20:30.036 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2023-07-10T17:20:30.036 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.c 2023-07-10T17:20:30.149 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:30 smithi116 systemd[1]: Stopping Ceph mon.c for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:20:30.644 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:30 smithi116 bash[65980]: debug 2023-07-10T17:20:30.246+0000 7fd4fa39a700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-07-10T17:20:30.645 INFO:journalctl@ceph.mon.c.smithi116.stdout:Jul 10 17:20:30 smithi116 bash[65980]: debug 2023-07-10T17:20:30.246+0000 7fd4fa39a700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2023-07-10T17:20:31.142 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.c.service' 2023-07-10T17:20:31.176 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-10T17:20:31.177 INFO:tasks.cephadm.mon.b:Stopped mon.c 2023-07-10T17:20:31.177 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2023-07-10T17:20:31.177 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.b 2023-07-10T17:20:31.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:31 smithi192 systemd[1]: Stopping Ceph mon.b for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:20:31.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:31 smithi192 bash[50362]: debug 2023-07-10T17:20:31.350+0000 7f9ecc921700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-07-10T17:20:31.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Jul 10 17:20:31 smithi192 bash[50362]: debug 2023-07-10T17:20:31.350+0000 7f9ecc921700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2023-07-10T17:20:32.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mon.b.service' 2023-07-10T17:20:32.462 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-10T17:20:32.463 INFO:tasks.cephadm.mon.b:Stopped mon.b 2023-07-10T17:20:32.464 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-10T17:20:32.465 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.y 2023-07-10T17:20:32.874 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:32 smithi116 systemd[1]: Stopping Ceph mgr.y for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:20:32.874 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:32 smithi116 bash[59394]: debug 2023-07-10T17:20:32.646+0000 7f6a12f1a700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:20:32.874 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Jul 10 17:20:32 smithi116 bash[59394]: debug 2023-07-10T17:20:32.646+0000 7f6a12f1a700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-07-10T17:20:33.568 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.y.service' 2023-07-10T17:20:33.601 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-10T17:20:33.602 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-10T17:20:33.602 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-10T17:20:33.603 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.x 2023-07-10T17:20:33.889 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Jul 10 17:20:33 smithi192 systemd[1]: Stopping Ceph mgr.x for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:20:34.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@mgr.x.service' 2023-07-10T17:20:34.634 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-10T17:20:34.635 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-10T17:20:34.635 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-10T17:20:34.636 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.0 2023-07-10T17:20:35.144 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:20:34 smithi116 systemd[1]: Stopping Ceph osd.0 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:20:35.145 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:20:34 smithi116 bash[70480]: debug 2023-07-10T17:20:34.842+0000 7f43f53c7700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:20:35.145 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:20:34 smithi116 bash[70480]: debug 2023-07-10T17:20:34.842+0000 7f43f53c7700 -1 osd.0 142 *** Got signal Terminated *** 2023-07-10T17:20:35.145 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:20:34 smithi116 bash[70480]: debug 2023-07-10T17:20:34.842+0000 7f43f53c7700 -1 osd.0 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-10T17:20:41.644 INFO:journalctl@ceph.osd.0.smithi116.stdout:Jul 10 17:20:41 smithi116 bash[92428]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-osd-0 2023-07-10T17:20:43.276 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.0.service' 2023-07-10T17:20:43.309 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-10T17:20:43.310 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-10T17:20:43.311 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-10T17:20:43.311 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.1 2023-07-10T17:20:43.531 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:20:43 smithi116 systemd[1]: Stopping Ceph osd.1 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:20:43.533 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:20:43 smithi116 bash[72834]: debug 2023-07-10T17:20:43.478+0000 7f16aeef7700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:20:43.533 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:20:43 smithi116 bash[72834]: debug 2023-07-10T17:20:43.478+0000 7f16aeef7700 -1 osd.1 142 *** Got signal Terminated *** 2023-07-10T17:20:43.534 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:20:43 smithi116 bash[72834]: debug 2023-07-10T17:20:43.478+0000 7f16aeef7700 -1 osd.1 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-10T17:20:43.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:20:43 smithi116 bash[88313]: ts=2023-07-10T17:20:43.527Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi116.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.116:8443: connect: connection refused" 2023-07-10T17:20:43.895 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:20:43 smithi116 bash[88313]: ts=2023-07-10T17:20:43.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi192.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.192:8443: connect: connection refused" 2023-07-10T17:20:49.894 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 10 17:20:49 smithi116 bash[92657]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-osd-1 2023-07-10T17:20:51.827 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.1.service' 2023-07-10T17:20:51.861 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-10T17:20:51.862 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-10T17:20:51.863 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-10T17:20:51.863 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.2 2023-07-10T17:20:52.144 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:20:51 smithi116 systemd[1]: Stopping Ceph osd.2 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:20:52.145 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:20:52 smithi116 bash[75175]: debug 2023-07-10T17:20:52.066+0000 7f2945890700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:20:52.145 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:20:52 smithi116 bash[75175]: debug 2023-07-10T17:20:52.066+0000 7f2945890700 -1 osd.2 142 *** Got signal Terminated *** 2023-07-10T17:20:52.145 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:20:52 smithi116 bash[75175]: debug 2023-07-10T17:20:52.066+0000 7f2945890700 -1 osd.2 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-10T17:20:53.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:20:53 smithi116 bash[88313]: ts=2023-07-10T17:20:53.525Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi116.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.116:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi192.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.192:8443: connect: connection refused" 2023-07-10T17:20:53.895 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:20:53 smithi116 bash[88313]: ts=2023-07-10T17:20:53.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi116.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.116:8443: connect: connection refused" 2023-07-10T17:20:53.895 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:20:53 smithi116 bash[88313]: ts=2023-07-10T17:20:53.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi192.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.192:8443: connect: connection refused" 2023-07-10T17:20:58.645 INFO:journalctl@ceph.osd.2.smithi116.stdout:Jul 10 17:20:58 smithi116 bash[92882]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-osd-2 2023-07-10T17:21:00.341 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.2.service' 2023-07-10T17:21:00.372 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-10T17:21:00.373 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-10T17:21:00.373 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-10T17:21:00.374 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.3 2023-07-10T17:21:00.644 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:21:00 smithi116 systemd[1]: Stopping Ceph osd.3 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:21:00.644 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:21:00 smithi116 bash[77480]: debug 2023-07-10T17:21:00.538+0000 7f2aa4556700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:21:00.645 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:21:00 smithi116 bash[77480]: debug 2023-07-10T17:21:00.538+0000 7f2aa4556700 -1 osd.3 142 *** Got signal Terminated *** 2023-07-10T17:21:00.645 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:21:00 smithi116 bash[77480]: debug 2023-07-10T17:21:00.538+0000 7f2aa4556700 -1 osd.3 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-10T17:21:00.886 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:00 smithi192 bash[59572]: debug 2023-07-10T17:21:00.609+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:01.635 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:01 smithi192 bash[57401]: debug 2023-07-10T17:21:01.185+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:01.886 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:01 smithi192 bash[59572]: debug 2023-07-10T17:21:01.625+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:02.597 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:02 smithi192 bash[57401]: debug 2023-07-10T17:21:02.189+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:02.598 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:02 smithi192 bash[52978]: debug 2023-07-10T17:21:02.401+0000 7f06f60a8700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.040240+0000 front 2023-07-10T17:20:38.040611+0000 (oldest deadline 2023-07-10T17:21:01.540021+0000) 2023-07-10T17:21:02.598 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:02 smithi192 bash[55213]: debug 2023-07-10T17:21:02.217+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:02.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:02 smithi192 bash[59572]: debug 2023-07-10T17:21:02.593+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:03.595 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:03 smithi192 bash[52978]: debug 2023-07-10T17:21:03.445+0000 7f06f60a8700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.040240+0000 front 2023-07-10T17:20:38.040611+0000 (oldest deadline 2023-07-10T17:21:01.540021+0000) 2023-07-10T17:21:03.596 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:03 smithi192 bash[55213]: debug 2023-07-10T17:21:03.253+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:03.597 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:03 smithi192 bash[57401]: debug 2023-07-10T17:21:03.197+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:03.886 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:03 smithi192 bash[59572]: debug 2023-07-10T17:21:03.589+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:03.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:03 smithi116 bash[88313]: ts=2023-07-10T17:21:03.526Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi116.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.116:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi192.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.192:8443: connect: connection refused" 2023-07-10T17:21:03.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:03 smithi116 bash[88313]: ts=2023-07-10T17:21:03.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi116.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.116:8443: connect: connection refused" 2023-07-10T17:21:03.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:03 smithi116 bash[88313]: ts=2023-07-10T17:21:03.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi192.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.192:8443: connect: connection refused" 2023-07-10T17:21:04.436 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:04 smithi192 bash[55213]: debug 2023-07-10T17:21:04.213+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:04.437 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:04 smithi192 bash[57401]: debug 2023-07-10T17:21:04.169+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:04.886 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:04 smithi192 bash[52978]: debug 2023-07-10T17:21:04.429+0000 7f06f60a8700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.040240+0000 front 2023-07-10T17:20:38.040611+0000 (oldest deadline 2023-07-10T17:21:01.540021+0000) 2023-07-10T17:21:04.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:04 smithi192 bash[59572]: debug 2023-07-10T17:21:04.597+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:05.465 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:05 smithi192 bash[55213]: debug 2023-07-10T17:21:05.205+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:05.466 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:05 smithi192 bash[57401]: debug 2023-07-10T17:21:05.157+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:05.887 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:05 smithi192 bash[52978]: debug 2023-07-10T17:21:05.461+0000 7f06f60a8700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.040240+0000 front 2023-07-10T17:20:38.040611+0000 (oldest deadline 2023-07-10T17:21:01.540021+0000) 2023-07-10T17:21:05.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:05 smithi192 bash[59572]: debug 2023-07-10T17:21:05.613+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:06.474 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:06 smithi192 bash[55213]: debug 2023-07-10T17:21:06.205+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:06.475 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:06 smithi192 bash[57401]: debug 2023-07-10T17:21:06.197+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:06.887 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:06 smithi192 bash[52978]: debug 2023-07-10T17:21:06.469+0000 7f06f60a8700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.040240+0000 front 2023-07-10T17:20:38.040611+0000 (oldest deadline 2023-07-10T17:21:01.540021+0000) 2023-07-10T17:21:06.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:06 smithi192 bash[59572]: debug 2023-07-10T17:21:06.649+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:07.144 INFO:journalctl@ceph.osd.3.smithi116.stdout:Jul 10 17:21:06 smithi116 bash[93108]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-osd-3 2023-07-10T17:21:07.462 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:07 smithi192 bash[55213]: debug 2023-07-10T17:21:07.181+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:07.463 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:07 smithi192 bash[57401]: debug 2023-07-10T17:21:07.173+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:07.887 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:07 smithi192 bash[52978]: debug 2023-07-10T17:21:07.457+0000 7f06f60a8700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.040240+0000 front 2023-07-10T17:20:38.040611+0000 (oldest deadline 2023-07-10T17:21:01.540021+0000) 2023-07-10T17:21:07.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:07 smithi192 bash[59572]: debug 2023-07-10T17:21:07.685+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:08.636 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:08 smithi192 bash[55213]: debug 2023-07-10T17:21:08.189+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:08.637 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:08 smithi192 bash[52978]: debug 2023-07-10T17:21:08.437+0000 7f06f60a8700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.040240+0000 front 2023-07-10T17:20:38.040611+0000 (oldest deadline 2023-07-10T17:21:01.540021+0000) 2023-07-10T17:21:08.638 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:08 smithi192 bash[57401]: debug 2023-07-10T17:21:08.213+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:08.817 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.3.service' 2023-07-10T17:21:08.848 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-10T17:21:08.849 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-10T17:21:08.849 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-10T17:21:08.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.4 2023-07-10T17:21:09.049 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:08 smithi192 systemd[1]: Stopping Ceph osd.4 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:21:09.050 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:08 smithi192 bash[59572]: debug 2023-07-10T17:21:08.657+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:09.388 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:09 smithi192 bash[52978]: debug 2023-07-10T17:21:09.045+0000 7f06faab4700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:21:09.389 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:09 smithi192 bash[52978]: debug 2023-07-10T17:21:09.045+0000 7f06faab4700 -1 osd.4 142 *** Got signal Terminated *** 2023-07-10T17:21:09.389 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:09 smithi192 bash[52978]: debug 2023-07-10T17:21:09.045+0000 7f06faab4700 -1 osd.4 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-10T17:21:09.390 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:09 smithi192 bash[57401]: debug 2023-07-10T17:21:09.209+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:09.390 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:09 smithi192 bash[57401]: debug 2023-07-10T17:21:09.209+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:09.390 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:09 smithi192 bash[55213]: debug 2023-07-10T17:21:09.197+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:09.669 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:09 smithi192 bash[52978]: debug 2023-07-10T17:21:09.389+0000 7f06f60a8700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.040240+0000 front 2023-07-10T17:20:38.040611+0000 (oldest deadline 2023-07-10T17:21:01.540021+0000) 2023-07-10T17:21:10.136 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:09 smithi192 bash[59572]: debug 2023-07-10T17:21:09.665+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:10.637 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:10 smithi192 bash[52978]: debug 2023-07-10T17:21:10.341+0000 7f06f60a8700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.040240+0000 front 2023-07-10T17:20:38.040611+0000 (oldest deadline 2023-07-10T17:21:01.540021+0000) 2023-07-10T17:21:10.649 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:10 smithi192 bash[55213]: debug 2023-07-10T17:21:10.249+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:10.650 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:10 smithi192 bash[57401]: debug 2023-07-10T17:21:10.237+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:10.650 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:10 smithi192 bash[57401]: debug 2023-07-10T17:21:10.237+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:11.137 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:10 smithi192 bash[59572]: debug 2023-07-10T17:21:10.689+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:11.137 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:10 smithi192 bash[59572]: debug 2023-07-10T17:21:10.689+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:11.637 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:11 smithi192 bash[52978]: debug 2023-07-10T17:21:11.309+0000 7f06f60a8700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.040240+0000 front 2023-07-10T17:20:38.040611+0000 (oldest deadline 2023-07-10T17:21:01.540021+0000) 2023-07-10T17:21:11.637 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:11 smithi192 bash[57401]: debug 2023-07-10T17:21:11.261+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:11.638 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:11 smithi192 bash[57401]: debug 2023-07-10T17:21:11.261+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:11.638 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:11 smithi192 bash[55213]: debug 2023-07-10T17:21:11.249+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:11.638 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:11 smithi192 bash[55213]: debug 2023-07-10T17:21:11.249+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:44.433087+0000 front 2023-07-10T17:20:44.433225+0000 (oldest deadline 2023-07-10T17:21:10.333022+0000) 2023-07-10T17:21:11.920 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:11 smithi192 bash[59572]: debug 2023-07-10T17:21:11.641+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:11.921 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:11 smithi192 bash[59572]: debug 2023-07-10T17:21:11.641+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:12.271 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:11 smithi192 bash[66523]: ts=2023-07-10T17:21:11.917Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.116:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.116:8765: connect: connection refused" 2023-07-10T17:21:12.272 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:11 smithi192 bash[66523]: ts=2023-07-10T17:21:11.922Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.116:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.116:8765: connect: connection refused" 2023-07-10T17:21:12.272 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:11 smithi192 bash[66523]: ts=2023-07-10T17:21:11.922Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.116:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.116:8765: connect: connection refused" 2023-07-10T17:21:12.272 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:11 smithi192 bash[66523]: ts=2023-07-10T17:21:11.924Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.116:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.116:8765: connect: connection refused" 2023-07-10T17:21:12.637 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:12 smithi192 bash[57401]: debug 2023-07-10T17:21:12.297+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:12.637 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:12 smithi192 bash[57401]: debug 2023-07-10T17:21:12.297+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:12.638 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:12 smithi192 bash[52978]: debug 2023-07-10T17:21:12.265+0000 7f06f60a8700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.040240+0000 front 2023-07-10T17:20:38.040611+0000 (oldest deadline 2023-07-10T17:21:01.540021+0000) 2023-07-10T17:21:12.638 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:12 smithi192 bash[52978]: debug 2023-07-10T17:21:12.265+0000 7f06f60a8700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:46.841799+0000 front 2023-07-10T17:20:46.841896+0000 (oldest deadline 2023-07-10T17:21:11.541305+0000) 2023-07-10T17:21:12.638 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:12 smithi192 bash[55213]: debug 2023-07-10T17:21:12.277+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:12.638 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:12 smithi192 bash[55213]: debug 2023-07-10T17:21:12.277+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:44.433087+0000 front 2023-07-10T17:20:44.433225+0000 (oldest deadline 2023-07-10T17:21:10.333022+0000) 2023-07-10T17:21:13.137 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:12 smithi192 bash[59572]: debug 2023-07-10T17:21:12.681+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:13.138 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:12 smithi192 bash[59572]: debug 2023-07-10T17:21:12.681+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:13.641 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:13 smithi192 bash[52978]: debug 2023-07-10T17:21:13.313+0000 7f06f60a8700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.040240+0000 front 2023-07-10T17:20:38.040611+0000 (oldest deadline 2023-07-10T17:21:01.540021+0000) 2023-07-10T17:21:13.641 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:13 smithi192 bash[52978]: debug 2023-07-10T17:21:13.313+0000 7f06f60a8700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:46.841799+0000 front 2023-07-10T17:20:46.841896+0000 (oldest deadline 2023-07-10T17:21:11.541305+0000) 2023-07-10T17:21:13.642 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:13 smithi192 bash[55213]: debug 2023-07-10T17:21:13.229+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:13.642 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:13 smithi192 bash[55213]: debug 2023-07-10T17:21:13.229+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:44.433087+0000 front 2023-07-10T17:20:44.433225+0000 (oldest deadline 2023-07-10T17:21:10.333022+0000) 2023-07-10T17:21:13.642 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:13 smithi192 bash[57401]: debug 2023-07-10T17:21:13.321+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:13.643 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:13 smithi192 bash[57401]: debug 2023-07-10T17:21:13.321+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:13.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:13 smithi116 bash[88313]: ts=2023-07-10T17:21:13.527Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=13 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi192.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.192:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi116.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.116:8443: connect: connection refused" 2023-07-10T17:21:13.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:13 smithi116 bash[88313]: ts=2023-07-10T17:21:13.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi116.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.116:8443: connect: connection refused" 2023-07-10T17:21:13.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:13 smithi116 bash[88313]: ts=2023-07-10T17:21:13.530Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi192.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.192:8443: connect: connection refused" 2023-07-10T17:21:14.136 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:13 smithi192 bash[59572]: debug 2023-07-10T17:21:13.641+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:14.137 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:13 smithi192 bash[59572]: debug 2023-07-10T17:21:13.641+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:14.608 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:14 smithi192 bash[55213]: debug 2023-07-10T17:21:14.277+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:14.608 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:14 smithi192 bash[55213]: debug 2023-07-10T17:21:14.277+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:44.433087+0000 front 2023-07-10T17:20:44.433225+0000 (oldest deadline 2023-07-10T17:21:10.333022+0000) 2023-07-10T17:21:14.609 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:14 smithi192 bash[57401]: debug 2023-07-10T17:21:14.365+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:14.609 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:14 smithi192 bash[57401]: debug 2023-07-10T17:21:14.365+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:14.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:14 smithi192 bash[59572]: debug 2023-07-10T17:21:14.601+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:14.889 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:14 smithi192 bash[59572]: debug 2023-07-10T17:21:14.601+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:15.633 INFO:journalctl@ceph.osd.4.smithi192.stdout:Jul 10 17:21:15 smithi192 bash[69910]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-osd-4 2023-07-10T17:21:15.633 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:15 smithi192 bash[55213]: debug 2023-07-10T17:21:15.321+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:15.634 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:15 smithi192 bash[55213]: debug 2023-07-10T17:21:15.321+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:44.433087+0000 front 2023-07-10T17:20:44.433225+0000 (oldest deadline 2023-07-10T17:21:10.333022+0000) 2023-07-10T17:21:15.634 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:15 smithi192 bash[57401]: debug 2023-07-10T17:21:15.401+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:15.634 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:15 smithi192 bash[57401]: debug 2023-07-10T17:21:15.401+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:15.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:15 smithi192 bash[59572]: debug 2023-07-10T17:21:15.629+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:15.889 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:15 smithi192 bash[59572]: debug 2023-07-10T17:21:15.629+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:16.610 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:16 smithi192 bash[55213]: debug 2023-07-10T17:21:16.333+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:16.611 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:16 smithi192 bash[55213]: debug 2023-07-10T17:21:16.333+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:44.433087+0000 front 2023-07-10T17:20:44.433225+0000 (oldest deadline 2023-07-10T17:21:10.333022+0000) 2023-07-10T17:21:16.611 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:16 smithi192 bash[57401]: debug 2023-07-10T17:21:16.441+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:16.611 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:16 smithi192 bash[57401]: debug 2023-07-10T17:21:16.441+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:16.884 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:16 smithi192 bash[59572]: debug 2023-07-10T17:21:16.605+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:16.884 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:16 smithi192 bash[59572]: debug 2023-07-10T17:21:16.605+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:17.589 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:17 smithi192 bash[55213]: debug 2023-07-10T17:21:17.309+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:17.590 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:17 smithi192 bash[55213]: debug 2023-07-10T17:21:17.309+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:44.433087+0000 front 2023-07-10T17:20:44.433225+0000 (oldest deadline 2023-07-10T17:21:10.333022+0000) 2023-07-10T17:21:17.590 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:17 smithi192 bash[57401]: debug 2023-07-10T17:21:17.421+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:17.590 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:17 smithi192 bash[57401]: debug 2023-07-10T17:21:17.421+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:17.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.4.service' 2023-07-10T17:21:17.627 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-10T17:21:17.628 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-10T17:21:17.628 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-10T17:21:17.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.5 2023-07-10T17:21:17.849 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:17 smithi192 systemd[1]: Stopping Ceph osd.5 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:21:17.850 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:17 smithi192 bash[59572]: debug 2023-07-10T17:21:17.597+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:17.850 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:17 smithi192 bash[59572]: debug 2023-07-10T17:21:17.597+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:18.137 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:17 smithi192 bash[55213]: debug 2023-07-10T17:21:17.845+0000 7f2f99e4c700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:21:18.137 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:17 smithi192 bash[55213]: debug 2023-07-10T17:21:17.845+0000 7f2f99e4c700 -1 osd.5 142 *** Got signal Terminated *** 2023-07-10T17:21:18.138 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:17 smithi192 bash[55213]: debug 2023-07-10T17:21:17.845+0000 7f2f99e4c700 -1 osd.5 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-10T17:21:18.590 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:18 smithi192 bash[55213]: debug 2023-07-10T17:21:18.293+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:18.590 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:18 smithi192 bash[55213]: debug 2023-07-10T17:21:18.293+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:44.433087+0000 front 2023-07-10T17:20:44.433225+0000 (oldest deadline 2023-07-10T17:21:10.333022+0000) 2023-07-10T17:21:18.591 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:18 smithi192 bash[57401]: debug 2023-07-10T17:21:18.469+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:18.591 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:18 smithi192 bash[57401]: debug 2023-07-10T17:21:18.469+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:18.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:18 smithi192 bash[59572]: debug 2023-07-10T17:21:18.585+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:18.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:18 smithi192 bash[59572]: debug 2023-07-10T17:21:18.585+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:19.622 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:19 smithi192 bash[55213]: debug 2023-07-10T17:21:19.285+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:19.622 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:19 smithi192 bash[55213]: debug 2023-07-10T17:21:19.285+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:44.433087+0000 front 2023-07-10T17:20:44.433225+0000 (oldest deadline 2023-07-10T17:21:10.333022+0000) 2023-07-10T17:21:19.622 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:19 smithi192 bash[55213]: debug 2023-07-10T17:21:19.285+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:55.034713+0000 front 2023-07-10T17:20:55.034739+0000 (oldest deadline 2023-07-10T17:21:18.534581+0000) 2023-07-10T17:21:19.623 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:19 smithi192 bash[57401]: debug 2023-07-10T17:21:19.445+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:19.623 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:19 smithi192 bash[57401]: debug 2023-07-10T17:21:19.445+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:19.886 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:19 smithi192 bash[59572]: debug 2023-07-10T17:21:19.617+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:19.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:19 smithi192 bash[59572]: debug 2023-07-10T17:21:19.617+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:20.602 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:20 smithi192 bash[57401]: debug 2023-07-10T17:21:20.417+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:20.603 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:20 smithi192 bash[57401]: debug 2023-07-10T17:21:20.417+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:20.603 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:20 smithi192 bash[55213]: debug 2023-07-10T17:21:20.253+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:20.603 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:20 smithi192 bash[55213]: debug 2023-07-10T17:21:20.253+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:44.433087+0000 front 2023-07-10T17:20:44.433225+0000 (oldest deadline 2023-07-10T17:21:10.333022+0000) 2023-07-10T17:21:20.604 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:20 smithi192 bash[55213]: debug 2023-07-10T17:21:20.253+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:55.034713+0000 front 2023-07-10T17:20:55.034739+0000 (oldest deadline 2023-07-10T17:21:18.534581+0000) 2023-07-10T17:21:20.886 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:20 smithi192 bash[59572]: debug 2023-07-10T17:21:20.597+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:20.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:20 smithi192 bash[59572]: debug 2023-07-10T17:21:20.597+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:21.611 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:21 smithi192 bash[57401]: debug 2023-07-10T17:21:21.461+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:21.612 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:21 smithi192 bash[57401]: debug 2023-07-10T17:21:21.461+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:21.612 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:21 smithi192 bash[57401]: debug 2023-07-10T17:21:21.461+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:56.015577+0000 front 2023-07-10T17:20:56.015528+0000 (oldest deadline 2023-07-10T17:21:21.315395+0000) 2023-07-10T17:21:21.612 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:21 smithi192 bash[55213]: debug 2023-07-10T17:21:21.297+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:21.613 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:21 smithi192 bash[55213]: debug 2023-07-10T17:21:21.297+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:44.433087+0000 front 2023-07-10T17:20:44.433225+0000 (oldest deadline 2023-07-10T17:21:10.333022+0000) 2023-07-10T17:21:21.613 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:21 smithi192 bash[55213]: debug 2023-07-10T17:21:21.297+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:55.034713+0000 front 2023-07-10T17:20:55.034739+0000 (oldest deadline 2023-07-10T17:21:18.534581+0000) 2023-07-10T17:21:21.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:21 smithi192 bash[59572]: debug 2023-07-10T17:21:21.605+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:21.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:21 smithi192 bash[59572]: debug 2023-07-10T17:21:21.605+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:22.604 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:22 smithi192 bash[57401]: debug 2023-07-10T17:21:22.489+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:22.605 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:22 smithi192 bash[57401]: debug 2023-07-10T17:21:22.489+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:22.605 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:22 smithi192 bash[57401]: debug 2023-07-10T17:21:22.489+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:56.015577+0000 front 2023-07-10T17:20:56.015528+0000 (oldest deadline 2023-07-10T17:21:21.315395+0000) 2023-07-10T17:21:22.605 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:22 smithi192 bash[55213]: debug 2023-07-10T17:21:22.257+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:38.031844+0000 front 2023-07-10T17:20:38.031681+0000 (oldest deadline 2023-07-10T17:21:02.131431+0000) 2023-07-10T17:21:22.606 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:22 smithi192 bash[55213]: debug 2023-07-10T17:21:22.257+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:44.433087+0000 front 2023-07-10T17:20:44.433225+0000 (oldest deadline 2023-07-10T17:21:10.333022+0000) 2023-07-10T17:21:22.606 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:22 smithi192 bash[55213]: debug 2023-07-10T17:21:22.257+0000 7f2f9343c700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:55.034713+0000 front 2023-07-10T17:20:55.034739+0000 (oldest deadline 2023-07-10T17:21:18.534581+0000) 2023-07-10T17:21:22.886 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:22 smithi192 bash[59572]: debug 2023-07-10T17:21:22.601+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:22.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:22 smithi192 bash[59572]: debug 2023-07-10T17:21:22.601+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:22.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:22 smithi192 bash[59572]: debug 2023-07-10T17:21:22.601+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:23.886 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:23 smithi192 bash[57401]: debug 2023-07-10T17:21:23.497+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:23.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:23 smithi192 bash[57401]: debug 2023-07-10T17:21:23.497+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:23.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:23 smithi192 bash[57401]: debug 2023-07-10T17:21:23.497+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:56.015577+0000 front 2023-07-10T17:20:56.015528+0000 (oldest deadline 2023-07-10T17:21:21.315395+0000) 2023-07-10T17:21:23.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:23 smithi192 bash[59572]: debug 2023-07-10T17:21:23.593+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:23.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:23 smithi192 bash[59572]: debug 2023-07-10T17:21:23.593+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:23.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:23 smithi192 bash[59572]: debug 2023-07-10T17:21:23.593+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:23.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:23 smithi116 bash[88313]: ts=2023-07-10T17:21:23.528Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=14 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi116.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.116:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi192.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.192:8443: connect: connection refused" 2023-07-10T17:21:23.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:23 smithi116 bash[88313]: ts=2023-07-10T17:21:23.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi116.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.116:8443: connect: connection refused" 2023-07-10T17:21:23.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:23 smithi116 bash[88313]: ts=2023-07-10T17:21:23.534Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi192.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.192:8443: connect: connection refused" 2023-07-10T17:21:24.473 INFO:journalctl@ceph.osd.5.smithi192.stdout:Jul 10 17:21:24 smithi192 bash[70141]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-osd-5 2023-07-10T17:21:24.638 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:24 smithi192 bash[57401]: debug 2023-07-10T17:21:24.469+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:24.638 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:24 smithi192 bash[57401]: debug 2023-07-10T17:21:24.469+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:24.639 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:24 smithi192 bash[57401]: debug 2023-07-10T17:21:24.469+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:56.015577+0000 front 2023-07-10T17:20:56.015528+0000 (oldest deadline 2023-07-10T17:21:21.315395+0000) 2023-07-10T17:21:24.639 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:24 smithi192 bash[59572]: debug 2023-07-10T17:21:24.577+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:24.639 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:24 smithi192 bash[59572]: debug 2023-07-10T17:21:24.577+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:24.639 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:24 smithi192 bash[59572]: debug 2023-07-10T17:21:24.577+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:24.640 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:24 smithi192 bash[59572]: debug 2023-07-10T17:21:24.577+0000 7f3f825fc700 -1 osd.7 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.54575.0:60 2.1d 2:bc7dff32:::realms.fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9.control:head [watch ping cookie 94560030535040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142) 2023-07-10T17:21:25.636 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:25 smithi192 bash[57401]: debug 2023-07-10T17:21:25.465+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:25.637 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:25 smithi192 bash[57401]: debug 2023-07-10T17:21:25.465+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:25.637 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:25 smithi192 bash[57401]: debug 2023-07-10T17:21:25.465+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:56.015577+0000 front 2023-07-10T17:20:56.015528+0000 (oldest deadline 2023-07-10T17:21:21.315395+0000) 2023-07-10T17:21:25.637 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:25 smithi192 bash[59572]: debug 2023-07-10T17:21:25.593+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:25.638 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:25 smithi192 bash[59572]: debug 2023-07-10T17:21:25.593+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:25.638 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:25 smithi192 bash[59572]: debug 2023-07-10T17:21:25.593+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:25.638 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:25 smithi192 bash[59572]: debug 2023-07-10T17:21:25.593+0000 7f3f825fc700 -1 osd.7 142 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.54575.0:60 2.1d 2:bc7dff32:::realms.fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9.control:head [watch ping cookie 94560030535040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142) 2023-07-10T17:21:26.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.5.service' 2023-07-10T17:21:26.555 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-10T17:21:26.556 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-10T17:21:26.556 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-10T17:21:26.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.6 2023-07-10T17:21:26.778 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:26 smithi192 bash[59572]: debug 2023-07-10T17:21:26.557+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:26.779 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:26 smithi192 bash[59572]: debug 2023-07-10T17:21:26.557+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:26.780 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:26 smithi192 bash[59572]: debug 2023-07-10T17:21:26.557+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:26.780 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:26 smithi192 bash[59572]: debug 2023-07-10T17:21:26.557+0000 7f3f825fc700 -1 osd.7 142 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.54575.0:60 2.1d 2:bc7dff32:::realms.fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9.control:head [watch ping cookie 94560030535040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142) 2023-07-10T17:21:26.780 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:26 smithi192 bash[57401]: debug 2023-07-10T17:21:26.481+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:26.781 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:26 smithi192 bash[57401]: debug 2023-07-10T17:21:26.481+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:26.781 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:26 smithi192 bash[57401]: debug 2023-07-10T17:21:26.481+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:56.015577+0000 front 2023-07-10T17:20:56.015528+0000 (oldest deadline 2023-07-10T17:21:21.315395+0000) 2023-07-10T17:21:26.781 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:26 smithi192 systemd[1]: Stopping Ceph osd.6 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:21:26.781 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:26 smithi192 bash[57401]: debug 2023-07-10T17:21:26.773+0000 7fb6d2bd3700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:21:26.781 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:26 smithi192 bash[57401]: debug 2023-07-10T17:21:26.773+0000 7fb6d2bd3700 -1 osd.6 142 *** Got signal Terminated *** 2023-07-10T17:21:27.136 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:26 smithi192 bash[57401]: debug 2023-07-10T17:21:26.773+0000 7fb6d2bd3700 -1 osd.6 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-10T17:21:27.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:27 smithi192 bash[57401]: debug 2023-07-10T17:21:27.481+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:27.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:27 smithi192 bash[57401]: debug 2023-07-10T17:21:27.481+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:27.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:27 smithi192 bash[57401]: debug 2023-07-10T17:21:27.481+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:56.015577+0000 front 2023-07-10T17:20:56.015528+0000 (oldest deadline 2023-07-10T17:21:21.315395+0000) 2023-07-10T17:21:27.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:27 smithi192 bash[59572]: debug 2023-07-10T17:21:27.589+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:27.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:27 smithi192 bash[59572]: debug 2023-07-10T17:21:27.589+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:27.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:27 smithi192 bash[59572]: debug 2023-07-10T17:21:27.589+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:27.889 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:27 smithi192 bash[59572]: debug 2023-07-10T17:21:27.589+0000 7f3f825fc700 -1 osd.7 142 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.54575.0:60 2.1d 2:bc7dff32:::realms.fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9.control:head [watch ping cookie 94560030535040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142) 2023-07-10T17:21:28.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:28 smithi192 bash[57401]: debug 2023-07-10T17:21:28.513+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:28.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:28 smithi192 bash[57401]: debug 2023-07-10T17:21:28.513+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:28.888 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:28 smithi192 bash[57401]: debug 2023-07-10T17:21:28.513+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:56.015577+0000 front 2023-07-10T17:20:56.015528+0000 (oldest deadline 2023-07-10T17:21:21.315395+0000) 2023-07-10T17:21:28.889 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:28 smithi192 bash[59572]: debug 2023-07-10T17:21:28.565+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:28.889 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:28 smithi192 bash[59572]: debug 2023-07-10T17:21:28.565+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:28.890 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:28 smithi192 bash[59572]: debug 2023-07-10T17:21:28.565+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:28.890 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:28 smithi192 bash[59572]: debug 2023-07-10T17:21:28.565+0000 7f3f825fc700 -1 osd.7 142 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.54575.0:60 2.1d 2:bc7dff32:::realms.fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9.control:head [watch ping cookie 94560030535040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142) 2023-07-10T17:21:29.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:29 smithi192 bash[57401]: debug 2023-07-10T17:21:29.501+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:29.888 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:29 smithi192 bash[57401]: debug 2023-07-10T17:21:29.501+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:29.888 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:29 smithi192 bash[57401]: debug 2023-07-10T17:21:29.501+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:56.015577+0000 front 2023-07-10T17:20:56.015528+0000 (oldest deadline 2023-07-10T17:21:21.315395+0000) 2023-07-10T17:21:29.888 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:29 smithi192 bash[57401]: debug 2023-07-10T17:21:29.501+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6830 osd.3 since back 2023-07-10T17:21:05.417203+0000 front 2023-07-10T17:21:05.417004+0000 (oldest deadline 2023-07-10T17:21:28.916845+0000) 2023-07-10T17:21:29.889 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:29 smithi192 bash[59572]: debug 2023-07-10T17:21:29.589+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:29.889 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:29 smithi192 bash[59572]: debug 2023-07-10T17:21:29.589+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:29.889 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:29 smithi192 bash[59572]: debug 2023-07-10T17:21:29.589+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:29.890 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:29 smithi192 bash[59572]: debug 2023-07-10T17:21:29.589+0000 7f3f825fc700 -1 osd.7 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.54575.0:60 2.1d 2:bc7dff32:::realms.fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9.control:head [watch ping cookie 94560030535040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142) 2023-07-10T17:21:30.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:30 smithi192 bash[57401]: debug 2023-07-10T17:21:30.481+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:30.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:30 smithi192 bash[57401]: debug 2023-07-10T17:21:30.481+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:30.888 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:30 smithi192 bash[57401]: debug 2023-07-10T17:21:30.481+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:56.015577+0000 front 2023-07-10T17:20:56.015528+0000 (oldest deadline 2023-07-10T17:21:21.315395+0000) 2023-07-10T17:21:30.888 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:30 smithi192 bash[57401]: debug 2023-07-10T17:21:30.481+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6830 osd.3 since back 2023-07-10T17:21:05.417203+0000 front 2023-07-10T17:21:05.417004+0000 (oldest deadline 2023-07-10T17:21:28.916845+0000) 2023-07-10T17:21:30.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:30 smithi192 bash[59572]: debug 2023-07-10T17:21:30.625+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:30.889 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:30 smithi192 bash[59572]: debug 2023-07-10T17:21:30.625+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:30.889 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:30 smithi192 bash[59572]: debug 2023-07-10T17:21:30.625+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:30.889 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:30 smithi192 bash[59572]: debug 2023-07-10T17:21:30.625+0000 7f3f825fc700 -1 osd.7 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.54575.0:60 2.1d 2:bc7dff32:::realms.fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9.control:head [watch ping cookie 94560030535040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142) 2023-07-10T17:21:31.887 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:31 smithi192 bash[57401]: debug 2023-07-10T17:21:31.497+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:36.311057+0000 front 2023-07-10T17:20:36.310917+0000 (oldest deadline 2023-07-10T17:21:00.410805+0000) 2023-07-10T17:21:31.888 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:31 smithi192 bash[57401]: debug 2023-07-10T17:21:31.497+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:47.413294+0000 front 2023-07-10T17:20:47.413317+0000 (oldest deadline 2023-07-10T17:21:09.112908+0000) 2023-07-10T17:21:31.889 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:31 smithi192 bash[57401]: debug 2023-07-10T17:21:31.497+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:56.015577+0000 front 2023-07-10T17:20:56.015528+0000 (oldest deadline 2023-07-10T17:21:21.315395+0000) 2023-07-10T17:21:31.889 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:31 smithi192 bash[57401]: debug 2023-07-10T17:21:31.497+0000 7fb6cd9c6700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.116:6830 osd.3 since back 2023-07-10T17:21:05.417203+0000 front 2023-07-10T17:21:05.417004+0000 (oldest deadline 2023-07-10T17:21:28.916845+0000) 2023-07-10T17:21:31.890 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:31 smithi192 bash[59572]: debug 2023-07-10T17:21:31.609+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:31.890 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:31 smithi192 bash[59572]: debug 2023-07-10T17:21:31.609+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:31.891 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:31 smithi192 bash[59572]: debug 2023-07-10T17:21:31.609+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:31.891 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:31 smithi192 bash[59572]: debug 2023-07-10T17:21:31.609+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6830 osd.3 since back 2023-07-10T17:21:05.175229+0000 front 2023-07-10T17:21:05.175246+0000 (oldest deadline 2023-07-10T17:21:31.075007+0000) 2023-07-10T17:21:31.891 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:31 smithi192 bash[59572]: debug 2023-07-10T17:21:31.613+0000 7f3f825fc700 -1 osd.7 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.54575.0:60 2.1d 2:bc7dff32:::realms.fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9.control:head [watch ping cookie 94560030535040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142) 2023-07-10T17:21:32.886 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:32 smithi192 bash[59572]: debug 2023-07-10T17:21:32.565+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:32.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:32 smithi192 bash[59572]: debug 2023-07-10T17:21:32.565+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:32.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:32 smithi192 bash[59572]: debug 2023-07-10T17:21:32.565+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:32.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:32 smithi192 bash[59572]: debug 2023-07-10T17:21:32.565+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6830 osd.3 since back 2023-07-10T17:21:05.175229+0000 front 2023-07-10T17:21:05.175246+0000 (oldest deadline 2023-07-10T17:21:31.075007+0000) 2023-07-10T17:21:32.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:32 smithi192 bash[59572]: debug 2023-07-10T17:21:32.565+0000 7f3f825fc700 -1 osd.7 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.54575.0:60 2.1d 2:bc7dff32:::realms.fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9.control:head [watch ping cookie 94560030535040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142) 2023-07-10T17:21:33.386 INFO:journalctl@ceph.osd.6.smithi192.stdout:Jul 10 17:21:33 smithi192 bash[70363]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-osd-6 2023-07-10T17:21:33.886 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:33 smithi192 bash[59572]: debug 2023-07-10T17:21:33.537+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:33.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:33 smithi192 bash[59572]: debug 2023-07-10T17:21:33.537+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:33.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:33 smithi192 bash[59572]: debug 2023-07-10T17:21:33.537+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:33.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:33 smithi192 bash[59572]: debug 2023-07-10T17:21:33.537+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6830 osd.3 since back 2023-07-10T17:21:05.175229+0000 front 2023-07-10T17:21:05.175246+0000 (oldest deadline 2023-07-10T17:21:31.075007+0000) 2023-07-10T17:21:33.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:33 smithi192 bash[59572]: debug 2023-07-10T17:21:33.537+0000 7f3f825fc700 -1 osd.7 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.54575.0:60 2.1d 2:bc7dff32:::realms.fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9.control:head [watch ping cookie 94560030535040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142) 2023-07-10T17:21:33.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:33 smithi116 bash[88313]: ts=2023-07-10T17:21:33.530Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=14 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi116.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.116:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi192.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.192:8443: connect: connection refused" 2023-07-10T17:21:33.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:33 smithi116 bash[88313]: ts=2023-07-10T17:21:33.532Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi116.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.116:8443: connect: connection refused" 2023-07-10T17:21:33.895 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:33 smithi116 bash[88313]: ts=2023-07-10T17:21:33.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi192.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.192:8443: connect: connection refused" 2023-07-10T17:21:34.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:34 smithi192 bash[59572]: debug 2023-07-10T17:21:34.520+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:34.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:34 smithi192 bash[59572]: debug 2023-07-10T17:21:34.520+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:34.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:34 smithi192 bash[59572]: debug 2023-07-10T17:21:34.520+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:34.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:34 smithi192 bash[59572]: debug 2023-07-10T17:21:34.520+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6830 osd.3 since back 2023-07-10T17:21:05.175229+0000 front 2023-07-10T17:21:05.175246+0000 (oldest deadline 2023-07-10T17:21:31.075007+0000) 2023-07-10T17:21:34.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:34 smithi192 bash[59572]: debug 2023-07-10T17:21:34.520+0000 7f3f825fc700 -1 osd.7 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.54575.0:60 2.1d 2:bc7dff32:::realms.fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9.control:head [watch ping cookie 94560030535040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142) 2023-07-10T17:21:35.358 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.6.service' 2023-07-10T17:21:35.389 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-10T17:21:35.389 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-10T17:21:35.390 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-10T17:21:35.390 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.7 2023-07-10T17:21:35.620 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:35 smithi192 systemd[1]: Stopping Ceph osd.7 for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:21:35.621 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:35 smithi192 bash[59572]: debug 2023-07-10T17:21:35.540+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:35.621 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:35 smithi192 bash[59572]: debug 2023-07-10T17:21:35.540+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:35.622 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:35 smithi192 bash[59572]: debug 2023-07-10T17:21:35.540+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:35.624 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:35 smithi192 bash[59572]: debug 2023-07-10T17:21:35.540+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6830 osd.3 since back 2023-07-10T17:21:05.175229+0000 front 2023-07-10T17:21:05.175246+0000 (oldest deadline 2023-07-10T17:21:31.075007+0000) 2023-07-10T17:21:35.624 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:35 smithi192 bash[59572]: debug 2023-07-10T17:21:35.540+0000 7f3f825fc700 -1 osd.7 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.54575.0:60 2.1d 2:bc7dff32:::realms.fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9.control:head [watch ping cookie 94560030535040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142) 2023-07-10T17:21:35.625 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:35 smithi192 bash[59572]: debug 2023-07-10T17:21:35.612+0000 7f3f87809700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-07-10T17:21:35.626 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:35 smithi192 bash[59572]: debug 2023-07-10T17:21:35.612+0000 7f3f87809700 -1 osd.7 142 *** Got signal Terminated *** 2023-07-10T17:21:35.626 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:35 smithi192 bash[59572]: debug 2023-07-10T17:21:35.612+0000 7f3f87809700 -1 osd.7 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-10T17:21:36.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:36 smithi192 bash[59572]: debug 2023-07-10T17:21:36.540+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:36.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:36 smithi192 bash[59572]: debug 2023-07-10T17:21:36.540+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:36.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:36 smithi192 bash[59572]: debug 2023-07-10T17:21:36.540+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:36.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:36 smithi192 bash[59572]: debug 2023-07-10T17:21:36.540+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6830 osd.3 since back 2023-07-10T17:21:05.175229+0000 front 2023-07-10T17:21:05.175246+0000 (oldest deadline 2023-07-10T17:21:31.075007+0000) 2023-07-10T17:21:36.889 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:36 smithi192 bash[59572]: debug 2023-07-10T17:21:36.540+0000 7f3f825fc700 -1 osd.7 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.54575.0:60 2.1d 2:bc7dff32:::realms.fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9.control:head [watch ping cookie 94560030535040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142) 2023-07-10T17:21:37.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:37 smithi192 bash[59572]: debug 2023-07-10T17:21:37.568+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:37.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:37 smithi192 bash[59572]: debug 2023-07-10T17:21:37.568+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:37.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:37 smithi192 bash[59572]: debug 2023-07-10T17:21:37.568+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:37.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:37 smithi192 bash[59572]: debug 2023-07-10T17:21:37.568+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6830 osd.3 since back 2023-07-10T17:21:05.175229+0000 front 2023-07-10T17:21:05.175246+0000 (oldest deadline 2023-07-10T17:21:31.075007+0000) 2023-07-10T17:21:37.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:37 smithi192 bash[59572]: debug 2023-07-10T17:21:37.568+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.192:6804 osd.4 since back 2023-07-10T17:21:11.576324+0000 front 2023-07-10T17:21:11.576432+0000 (oldest deadline 2023-07-10T17:21:36.876255+0000) 2023-07-10T17:21:37.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:37 smithi192 bash[59572]: debug 2023-07-10T17:21:37.568+0000 7f3f825fc700 -1 osd.7 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.54575.0:60 2.1d 2:bc7dff32:::realms.fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9.control:head [watch ping cookie 94560030535040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142) 2023-07-10T17:21:38.886 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:38 smithi192 bash[59572]: debug 2023-07-10T17:21:38.576+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:38.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:38 smithi192 bash[59572]: debug 2023-07-10T17:21:38.576+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:38.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:38 smithi192 bash[59572]: debug 2023-07-10T17:21:38.576+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:38.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:38 smithi192 bash[59572]: debug 2023-07-10T17:21:38.576+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6830 osd.3 since back 2023-07-10T17:21:05.175229+0000 front 2023-07-10T17:21:05.175246+0000 (oldest deadline 2023-07-10T17:21:31.075007+0000) 2023-07-10T17:21:38.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:38 smithi192 bash[59572]: debug 2023-07-10T17:21:38.576+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.192:6804 osd.4 since back 2023-07-10T17:21:11.576324+0000 front 2023-07-10T17:21:11.576432+0000 (oldest deadline 2023-07-10T17:21:36.876255+0000) 2023-07-10T17:21:38.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:38 smithi192 bash[59572]: debug 2023-07-10T17:21:38.576+0000 7f3f825fc700 -1 osd.7 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.54575.0:60 2.1d 2:bc7dff32:::realms.fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9.control:head [watch ping cookie 94560030535040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142) 2023-07-10T17:21:39.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:39 smithi192 bash[59572]: debug 2023-07-10T17:21:39.588+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2023-07-10T17:20:35.865728+0000 front 2023-07-10T17:20:35.865508+0000 (oldest deadline 2023-07-10T17:21:00.565430+0000) 2023-07-10T17:21:39.887 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:39 smithi192 bash[59572]: debug 2023-07-10T17:21:39.588+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6814 osd.1 since back 2023-07-10T17:20:43.967646+0000 front 2023-07-10T17:20:43.967815+0000 (oldest deadline 2023-07-10T17:21:09.867337+0000) 2023-07-10T17:21:39.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:39 smithi192 bash[59572]: debug 2023-07-10T17:21:39.588+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6822 osd.2 since back 2023-07-10T17:20:54.871557+0000 front 2023-07-10T17:20:54.871511+0000 (oldest deadline 2023-07-10T17:21:21.872199+0000) 2023-07-10T17:21:39.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:39 smithi192 bash[59572]: debug 2023-07-10T17:21:39.588+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.116:6830 osd.3 since back 2023-07-10T17:21:05.175229+0000 front 2023-07-10T17:21:05.175246+0000 (oldest deadline 2023-07-10T17:21:31.075007+0000) 2023-07-10T17:21:39.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:39 smithi192 bash[59572]: debug 2023-07-10T17:21:39.588+0000 7f3f825fc700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.192:6804 osd.4 since back 2023-07-10T17:21:11.576324+0000 front 2023-07-10T17:21:11.576432+0000 (oldest deadline 2023-07-10T17:21:36.876255+0000) 2023-07-10T17:21:39.888 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:39 smithi192 bash[59572]: debug 2023-07-10T17:21:39.588+0000 7f3f825fc700 -1 osd.7 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.54575.0:60 2.1d 2:bc7dff32:::realms.fc7ea6ec-a545-48d6-8d3a-4651d0ffeeb9.control:head [watch ping cookie 94560030535040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142) 2023-07-10T17:21:42.137 INFO:journalctl@ceph.osd.7.smithi192.stdout:Jul 10 17:21:41 smithi192 bash[70593]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-osd-7 2023-07-10T17:21:43.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@osd.7.service' 2023-07-10T17:21:43.809 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-10T17:21:43.810 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-10T17:21:43.811 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2023-07-10T17:21:43.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a 2023-07-10T17:21:43.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:43 smithi116 bash[88313]: ts=2023-07-10T17:21:43.530Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=14 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi116.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.116:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi192.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.192:8443: connect: connection refused" 2023-07-10T17:21:43.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:43 smithi116 bash[88313]: ts=2023-07-10T17:21:43.532Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi116.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.116:8443: connect: connection refused" 2023-07-10T17:21:43.894 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:43 smithi116 bash[88313]: ts=2023-07-10T17:21:43.534Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi192.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.192:8443: connect: connection refused" 2023-07-10T17:21:44.063 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:43 smithi192 systemd[1]: Stopping Ceph prometheus.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:21:44.387 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:44 smithi192 bash[66523]: ts=2023-07-10T17:21:44.060Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2023-07-10T17:21:44.388 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:44 smithi192 bash[66523]: ts=2023-07-10T17:21:44.060Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2023-07-10T17:21:44.388 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:44 smithi192 bash[66523]: ts=2023-07-10T17:21:44.060Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2023-07-10T17:21:44.389 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:44 smithi192 bash[66523]: ts=2023-07-10T17:21:44.060Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2023-07-10T17:21:44.389 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:44 smithi192 bash[66523]: ts=2023-07-10T17:21:44.060Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2023-07-10T17:21:44.389 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:44 smithi192 bash[66523]: ts=2023-07-10T17:21:44.060Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2023-07-10T17:21:44.389 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:44 smithi192 bash[66523]: ts=2023-07-10T17:21:44.060Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2023-07-10T17:21:44.389 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:44 smithi192 bash[66523]: ts=2023-07-10T17:21:44.060Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2023-07-10T17:21:44.390 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:44 smithi192 bash[66523]: ts=2023-07-10T17:21:44.061Z caller=main.go:907 level=info msg="Scrape manager stopped" 2023-07-10T17:21:44.390 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:44 smithi192 bash[66523]: ts=2023-07-10T17:21:44.062Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2023-07-10T17:21:44.390 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:44 smithi192 bash[66523]: ts=2023-07-10T17:21:44.062Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2023-07-10T17:21:44.392 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Jul 10 17:21:44 smithi192 bash[66523]: ts=2023-07-10T17:21:44.062Z caller=main.go:1147 level=info msg="See you next time!" 2023-07-10T17:21:44.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@prometheus.a.service' 2023-07-10T17:21:44.798 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-10T17:21:44.799 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2023-07-10T17:21:44.799 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c --force --keep-logs 2023-07-10T17:21:44.961 INFO:teuthology.orchestra.run.smithi116.stderr:usage: cephadm [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR] 2023-07-10T17:21:44.962 INFO:teuthology.orchestra.run.smithi116.stderr: [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] 2023-07-10T17:21:44.962 INFO:teuthology.orchestra.run.smithi116.stderr: [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT] 2023-07-10T17:21:44.963 INFO:teuthology.orchestra.run.smithi116.stderr: [--retry RETRY] [--env ENV] [--no-container-init] 2023-07-10T17:21:44.963 INFO:teuthology.orchestra.run.smithi116.stderr: {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,exporter,host-maintenance,verify-prereqs} 2023-07-10T17:21:44.963 INFO:teuthology.orchestra.run.smithi116.stderr: ... 2023-07-10T17:21:44.963 INFO:teuthology.orchestra.run.smithi116.stderr:cephadm: error: unrecognized arguments: --keep-logs 2023-07-10T17:21:44.976 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-07-10T17:21:44.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c --force --keep-logs 2023-07-10T17:21:45.137 INFO:teuthology.orchestra.run.smithi192.stderr:usage: cephadm [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR] 2023-07-10T17:21:45.137 INFO:teuthology.orchestra.run.smithi192.stderr: [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] 2023-07-10T17:21:45.138 INFO:teuthology.orchestra.run.smithi192.stderr: [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT] 2023-07-10T17:21:45.138 INFO:teuthology.orchestra.run.smithi192.stderr: [--retry RETRY] [--env ENV] [--no-container-init] 2023-07-10T17:21:45.138 INFO:teuthology.orchestra.run.smithi192.stderr: {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,exporter,host-maintenance,verify-prereqs} 2023-07-10T17:21:45.138 INFO:teuthology.orchestra.run.smithi192.stderr: ... 2023-07-10T17:21:45.138 INFO:teuthology.orchestra.run.smithi192.stderr:cephadm: error: unrecognized arguments: --keep-logs 2023-07-10T17:21:45.153 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-07-10T17:21:45.154 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-10T17:21:45.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-10T17:21:45.185 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-10T17:21:45.188 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2023-07-10_14:39:39-orch-wip-yuri8-testing-2023-07-06-0909-reef-distro-default-smithi/7332179/remote/smithi116/crash 2023-07-10T17:21:45.190 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/crash -- . 2023-07-10T17:21:45.228 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2023-07-10_14:39:39-orch-wip-yuri8-testing-2023-07-06-0909-reef-distro-default-smithi/7332179/remote/smithi192/crash 2023-07-10T17:21:45.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/crash -- . 2023-07-10T17:21:45.245 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-10T17:21:45.246 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-07-10T17:21:45.262 INFO:teuthology.orchestra.run.smithi116.stderr:grep: /var/log/ceph/3f83c2ec-1f42-11ee-9b31-001a4aab830c/ceph.log: No such file or directory 2023-07-10T17:21:45.264 INFO:tasks.cephadm:Compressing logs... 2023-07-10T17:21:45.265 DEBUG:teuthology.orchestra.run.smithi116:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-10T17:21:45.267 DEBUG:teuthology.orchestra.run.smithi192:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-10T17:21:45.276 INFO:teuthology.orchestra.run.smithi116.stderr:find: '/var/log/rbd-target-api': No such file or directory 2023-07-10T17:21:45.307 INFO:teuthology.orchestra.run.smithi192.stderr:find: '/var/log/rbd-target-api': No such file or directory 2023-07-10T17:21:52.589 INFO:tasks.cephadm:Archiving logs... 2023-07-10T17:21:52.589 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-07-10_14:39:39-orch-wip-yuri8-testing-2023-07-06-0909-reef-distro-default-smithi/7332179/remote/smithi116/log 2023-07-10T17:21:52.591 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-10T17:21:53.516 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-07-10_14:39:39-orch-wip-yuri8-testing-2023-07-06-0909-reef-distro-default-smithi/7332179/remote/smithi192/log 2023-07-10T17:21:53.517 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-10T17:21:53.900 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:53 smithi116 bash[88313]: ts=2023-07-10T17:21:53.531Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=14 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi192.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.192:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi116.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.116:8443: connect: connection refused" 2023-07-10T17:21:53.900 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:53 smithi116 bash[88313]: ts=2023-07-10T17:21:53.534Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi116.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.116:8443: connect: connection refused" 2023-07-10T17:21:53.900 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:21:53 smithi116 bash[88313]: ts=2023-07-10T17:21:53.535Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi192.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.192:8443: connect: connection refused" 2023-07-10T17:21:54.280 INFO:tasks.cephadm:Removing cluster... 2023-07-10T17:21:54.281 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c --force 2023-07-10T17:21:59.108 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:21:58 smithi116 systemd[1]: Stopping Ceph node-exporter.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:22:00.144 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:21:59 smithi116 bash[93562]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-node-exporter-a 2023-07-10T17:22:00.144 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:21:59 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2023-07-10T17:22:00.144 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:21:59 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@node-exporter.a.service: Failed with result 'exit-code'. 2023-07-10T17:22:00.145 INFO:journalctl@ceph.node-exporter.a.smithi116.stdout:Jul 10 17:21:59 smithi116 systemd[1]: Stopped Ceph node-exporter.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:22:00.452 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:22:00 smithi116 systemd[1]: Stopping Ceph alertmanager.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:22:00.453 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:22:00 smithi116 bash[88313]: ts=2023-07-10T17:22:00.375Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2023-07-10T17:22:01.394 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:22:01 smithi116 bash[93701]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-alertmanager-a 2023-07-10T17:22:01.394 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:22:01 smithi116 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@alertmanager.a.service: Succeeded. 2023-07-10T17:22:01.395 INFO:journalctl@ceph.alertmanager.a.smithi116.stdout:Jul 10 17:22:01 smithi116 systemd[1]: Stopped Ceph alertmanager.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:22:36.176 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c --force 2023-07-10T17:22:40.841 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:22:40 smithi192 systemd[1]: Stopping Ceph node-exporter.b for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:22:41.636 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:22:41 smithi192 bash[71159]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-node-exporter-b 2023-07-10T17:22:41.637 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:22:41 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2023-07-10T17:22:41.637 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:22:41 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@node-exporter.b.service: Failed with result 'exit-code'. 2023-07-10T17:22:41.637 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Jul 10 17:22:41 smithi192 systemd[1]: Stopped Ceph node-exporter.b for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:22:42.031 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:22:41 smithi192 systemd[1]: Stopping Ceph grafana.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c... 2023-07-10T17:22:42.798 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:22:42 smithi192 bash[71305]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c-grafana-a 2023-07-10T17:22:42.799 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:22:42 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@grafana.a.service: Main process exited, code=exited, status=143/n/a 2023-07-10T17:22:42.799 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:22:42 smithi192 systemd[1]: ceph-3f83c2ec-1f42-11ee-9b31-001a4aab830c@grafana.a.service: Failed with result 'exit-code'. 2023-07-10T17:22:42.799 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Jul 10 17:22:42 smithi192 systemd[1]: Stopped Ceph grafana.a for 3f83c2ec-1f42-11ee-9b31-001a4aab830c. 2023-07-10T17:23:05.628 INFO:tasks.cephadm:Removing cephadm ... 2023-07-10T17:23:05.628 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-10T17:23:05.636 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-10T17:23:05.641 INFO:tasks.cephadm:Teardown complete 2023-07-10T17:23:05.642 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-10T17:23:05.656 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-10T17:23:05.657 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-10T17:23:05.680 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-10T17:23:05.698 INFO:teuthology.orchestra.run.smithi116.stdout: remote refid st t when poll reach delay offset jitter 2023-07-10T17:23:05.698 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================== 2023-07-10T17:23:05.699 INFO:teuthology.orchestra.run.smithi116.stdout:*hv01.front.sepi 44.4.53.2 2 u 24 64 377 0.066 3.219 0.243 2023-07-10T17:23:05.699 INFO:teuthology.orchestra.run.smithi116.stdout:+hv02.front.sepi 63.231.80.2 3 u 25 64 377 0.095 4.396 0.246 2023-07-10T17:23:05.699 INFO:teuthology.orchestra.run.smithi116.stdout:+hv03.front.sepi 74.6.168.72 3 u 16 64 377 0.092 0.365 0.486 2023-07-10T17:23:05.699 INFO:teuthology.orchestra.run.smithi116.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-07-10T17:23:05.701 INFO:teuthology.orchestra.run.smithi192.stdout: remote refid st t when poll reach delay offset jitter 2023-07-10T17:23:05.701 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================== 2023-07-10T17:23:05.702 INFO:teuthology.orchestra.run.smithi192.stdout:*hv01.front.sepi 44.4.53.2 2 u 29 64 377 0.097 2.026 0.362 2023-07-10T17:23:05.702 INFO:teuthology.orchestra.run.smithi192.stdout:+hv02.front.sepi 63.231.80.2 3 u 32 64 377 0.080 3.198 0.171 2023-07-10T17:23:05.702 INFO:teuthology.orchestra.run.smithi192.stdout:+hv03.front.sepi 74.6.168.72 3 u 29 64 377 0.101 -1.534 0.362 2023-07-10T17:23:05.702 INFO:teuthology.orchestra.run.smithi192.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-07-10T17:23:05.703 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-10T17:23:05.714 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-10T17:23:05.715 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-10T17:23:05.730 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-10T17:23:05.765 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-10T17:23:05.797 INFO:teuthology.task.internal:Duration was 2156.527524 seconds 2023-07-10T17:23:05.798 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-10T17:23:05.837 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-10T17:23:05.838 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-10T17:23:05.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-10T17:23:05.892 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-10T17:23:05.892 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2023-07-10T17:23:05.893 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-10T17:23:05.905 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2023-07-10T17:23:05.905 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-10T17:23:05.920 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-10T17:23:05.921 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-10T17:23:05.953 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-10T17:23:06.026 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-10T17:23:06.027 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-10T17:23:06.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-10T17:23:06.198 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-10T17:23:06.211 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-10T17:23:06.212 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-10T17:23:06.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-10T17:23:06.245 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-10T17:23:06.260 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-10T17:23:06.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-10T17:23:06.288 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2023-07-10T17:23:06.307 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2023-07-10T17:23:06.334 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-10T17:23:06.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-10T17:23:06.340 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-10T17:23:06.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-10T17:23:06.347 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-10T17:23:06.358 INFO:teuthology.task.internal:Transferring archived files... 2023-07-10T17:23:06.360 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-07-10_14:39:39-orch-wip-yuri8-testing-2023-07-06-0909-reef-distro-default-smithi/7332179/remote/smithi116 2023-07-10T17:23:06.361 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-10T17:23:06.390 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-07-10_14:39:39-orch-wip-yuri8-testing-2023-07-06-0909-reef-distro-default-smithi/7332179/remote/smithi192 2023-07-10T17:23:06.391 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-10T17:23:06.421 INFO:teuthology.task.internal:Removing archive directory... 2023-07-10T17:23:06.422 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-10T17:23:06.432 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-10T17:23:06.466 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-10T17:23:06.479 INFO:teuthology.task.internal:Not uploading archives. 2023-07-10T17:23:06.479 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-10T17:23:06.500 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-10T17:23:06.501 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-10T17:23:06.503 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-10T17:23:06.509 INFO:teuthology.orchestra.run.smithi116.stdout: 924113 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 10 17:23 /home/ubuntu/cephtest 2023-07-10T17:23:06.511 INFO:teuthology.orchestra.run.smithi192.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 10 17:23 /home/ubuntu/cephtest 2023-07-10T17:23:06.514 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-10T17:23:06.534 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-10T17:23:06.599 INFO:teuthology.nuke:Checking targets against current locks 2023-07-10T17:23:06.633 DEBUG:teuthology.nuke:shortname: smithi116 2023-07-10T17:23:06.634 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-10T17:23:06.657 DEBUG:teuthology.nuke:shortname: smithi192 2023-07-10T17:23:06.658 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-10T17:23:06.663 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-10_14:39:39-orch-wip-yuri8-testing-2023-07-06-0909-reef-distro-default-smithi/7332179', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-10 16:38:16.446351', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='} 2023-07-10T17:23:06.680 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-10_14:39:39-orch-wip-yuri8-testing-2023-07-06-0909-reef-distro-default-smithi/7332179', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-10 16:38:16.444956', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='} 2023-07-10T17:23:06.722 INFO:teuthology.orchestra.console:Power off smithi116 2023-07-10T17:23:06.724 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-10T17:23:06.751 INFO:teuthology.orchestra.console:Power off smithi192 2023-07-10T17:23:06.752 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-10T17:23:06.767 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-10T17:23:06.785 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-10T17:23:10.800 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-10T17:23:10.811 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-10T17:23:14.916 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-10T17:23:14.929 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-10T17:23:15.145 INFO:teuthology.orchestra.console:Power off for smithi192 completed 2023-07-10T17:23:19.036 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-10T17:23:19.248 INFO:teuthology.orchestra.console:Power off for smithi116 completed 2023-07-10T17:23:19.354 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} duration: 2156.5275242328644 failure_reason: 'Command failed on smithi116 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f83c2ec-1f42-11ee-9b31-001a4aab830c -- bash -c ''ceph orch upgrade ls''"' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c0ea79b85234479bb1a232c51a6d93ad status: fail success: false 2023-07-10T17:23:19.355 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-10T17:23:19.443 INFO:teuthology.run:FAIL